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EXECUTIVE  SUMMARY 


The  Ada  community,  including  government,  industry,  and  academic  personnel, 
needs  the  capability  to  assess  APSEs  (Ada  Programming  Support  Environments)  and  their 
components,  and  to  determine  their  conformance  to  applicable  standards  (e.g.,  DoD- 
STD-1838,  the  CAIS  standard).  The  technology  required  to  fully  satisfy  this  need  is  exten¬ 
sive  and  largely  unavailable;  it  cannot  be  acquired  by  a  single  government-sponsored, 
professional  society-sponsored,  or  private  effort.  The  purpose  of  the  APSE  Evaluation  and 
Validation  (E&V)  Task  is  to  provide  a  focai  point  for  addressing  the  need  by:  ' 

(1)  Identifying  and  defining  specific  technology  requirements, 

(2)  Developing  selected  elements  of  this  technology, 

(3)  Encouraging  others  to  develop  additional  elements!  and 

(4)  Collecting  information  describing  elements  which  already  exist. 

This  information  will  be  made  available  to  DoD  components,  other  government  agencies, 
industry  and  academia. 

The  purpose  of  the  E&V  Guidebook  (this  document)  is  to  provide  information 
that  will  help  users  to  assess  APSEs  and  APSE  components  by: 

"f  1 )  Assisting  in  the  selection  of  E&V  procedures,  the  interpretation  of 
resuits,  and  integration  of  analyses  and  results', 

(2)  Describing  E&V  procedures  and  techniques  developed  by  the  E&V 
Task,  and 

’  (3)  Assisting  in  the  location  of  E&V  procedures  and  techniques  devel¬ 
oped  outside  the  E&V  Task. 

All  E&V  procedures  and  techniques  found  in  the  Guidebook  are  referenced  by  the  indexes 
contained  in  the  companion  document  called  the  E&V  Reference  Manual. 
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Chapter  1  through  4  provide  a  general  introduction  to  the  document  and  other 
background  material.  Chapter  5  and  later  chapters  are  “formal  chapters”  built  around  a 
standard  format  and  formal  grammar.  Each  of  the  formal  chapters  contains  all  the  assess¬ 
ment  procedures  and  techniques  associated  with  a  particular  group  of  tools  or  toolsets  to 
be  assessed,  such  as  Compilation  System  Assessors  or  Test  System  Assessors.  The  as¬ 
sessment  procedures  are  described  and  in  some  instances  can  be  applied  directly  from 
the  information  given  in  the  Guidebook.  In  other  cases,  the  user  is  directed  to  a  primary 
reference  for  more  information. 

Yearly  updates  and  extensions  to  this  document  are  planned.  Therefore,  com¬ 
ments  and  suggestions  are  welcome.  Please  send  comments  electronically  (preferred)  to 
szymansk@a;po. sei.cmu.edu,  or  by  regular  mail  to  Mr.  Raymond  Szymanski,  AFWAL/AAAF, 
Wright  Patterson  AFB,  QH  45433-6543. 


I 

□ 


Tv _ 

r  •'  - 

A-.'-:  •'  '  • 


Note: 

"E&V  Guidebook,  Version  1.1”  (this  document)  is  best  used  in  conjunction  with  “E&V 
Reference  Manual,  Version  1.1.”  The  table  on  the  following  page  is  included  for  the 
benefit  of  readers  who  have  the  older  manual,  "E&V  Reference  Manual,  Version  1.0,” 
The  organization  of  the  Guidebook  was  significantly  altered  after  issuance  of  Version 
1.0  of  the  Reference  Manual.  (There  is  no  Guidebook,  Version  1.0.) 
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1.  INTRODUCTION 


1.1  PURPOSE  OF  GUIDEBOOK 

This  document  is  a  product  of  the  Ada  Programming  Support  Environment 
(APSE)  Evaluation  and  Validation  (E&V)  Task  sponsored  by  the  Ada  Joint  Program  Office. 
It  is  one  of  a  pair  of  companion  documents  known  as  the  E&V  Reference  System,  consist¬ 
ing  of: 


•  E&V  Reference  Manual 

•  E&V  Guidebook. 

The  subject  of  both  documents  is  the  assessment  of  APSEs  and  their  components.  Spe¬ 
cific  assessment  techniques  typically  fall  into  one  of  two  categories:  evaluation  (assess¬ 
ment  of  performance  and  quality)  and  validation  (assessment  of  conformance  to  a 
standard). 


The  purpose  of  the  Guidebook  is  to  provide  a  collection  of  information  to  sup¬ 
port  a  variety  of  E&V  users  in  the  following  ways.  It  should  help  them: 


•  Gain  an  overall  understanding  of  APSE  assessment,  in  particular, 
the  selection  of  appropriate  E&V  procedures,  the  interpretation  of 
test  results,  and  the  integration  of  analyses  and  results. 

•  Apply  the  various  E&V  procedures  and  techniques  developed  un¬ 
der  E&V  Task  sponsorship. 

•  Find  the  primary  sources  for  those  E&V  procedures  and  tech¬ 
niques  not  developed  by  the  E&V  Task  or  not  fully  explained  within 
the  Guidebook  (due  to  space  or  other  constraints). 


The  Re  -,/ce  Manual  includes  many  "pointers”  to  sections  in  the  Guidebook  and  other 
documents  which  describe  E&V  techniques  in  much  the  same  way  that  a  card  catalog 
does  in  a  library.  Figure  1.1-1  illustrates  the  relationship  between  the  documents. 
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1.2  THE  NEED  FOR  E&V  TECHNOLOGY 

Technology  for  the  assessment  of  APSEs  and  APSE  components  (toots)  is 
needed  because  of  the  difficulty  in  assessing  APSEs  and  because  of  the  importance  of  the 
decisions  made  based  on  these  assessments.  The  importance  of  an  APSE  selection  is 
evident  when  one  considers  the  large,  critical,  Ada-based  systems  to  be  developed  in  the 
coming  years.  The  effectiveness,  reliability,  and  cost  of  these  systems  will  be  strongly  in¬ 
fluenced  by  the  environments  used  to  develop  and  maintain  them.  From  the  point  of  view 
of  a  software  developing  organization,  the  decision  to  select  an  APSE  can  be  an  important 
investment  decision  with  long-lasting  influence  on  a  number  of  projects  and  the  organiza¬ 
tion's  methods  of  operation,  training,  and  competitiveness.  From  the  point  of  view  of  a 
software  maintenance  organization,  the  environment  used  will  strongly  influence  the  organi¬ 
zation's  effectiveness,  as  well  as  the  cost  of  its  operations  and  training. 

The  difficulty  of  assessing  APSEs  and  tools  exists  for  several  reasons.  First, 
an  APSE  represents  very  complex  technology  with  many  elements,  which  can  be  assessed 
individually  or  in  combination.  Second  there  is  a  confusing  diversity  of  choice  with  respect 
to  individual  tools,  tool  sets,  or  “whole  APSEs";  and  there  are  a  number  of  ways  of  view¬ 
ing  APSEs;  see  Chapter  3  of  the  E&V  Reference  Manual  [@RM  3].*  Third,  the  state  of  the 
art  of  APSE  architecture  and  of  some  categories  of  tools  (e.g.,  graphic  design  tools)  is  un¬ 
dergoing  rapid  change.  Finally,  there  is  a  lack  of  historical  data  relevant  to  APSEs,  partly 
because  of  the  general  pace  of  technological  change  and  partly  because  we  are  dealing 
with  Ada,  a  relatively  new  implementation  language.  E&V  technology  provides  methods  and 
techniques  to  overcome  these  difficulties  and  provides  a  basis  for  determining  performance 
and  other  attributes  of  APSEs. 

In  addition  to  the  need  for  assessment  technology  itself,  there  is  a  need  for  in¬ 
formation  about  this  technology.  Potential  buyers  and  users  of  APSEs  and  tools  need  a 
framework  for  understanding  APSEs  and  their  assessment,  as  well  as  information  about 
specific  assessment  techniques.  Similarly,  vendors  of  tools  and  APSEs  need  to  be  aware 
of  the  deficiencies  of  current  products,  as  well  as  the  criteria  to  be  used  in  the  assessment 
of  future  products.  Such  awareness  on  both  sides,  expressed  in  a  common  terminology, 
should  speed  up  the  evolution  of  tetter  software  development  environments. 

'The  format  used  for  references  is  associated  with  the  “formal  grammar"  used  beginning 
with  Chapter  5.  See  further  explanations  in  Appendix  C. 
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1.3  BACKGROUND 

In  June  1983  the  Ada  Joint  Program  Office  (AJPO)  proposed  the  formation  of 
the  E&V  Task  and  a  tri-service  E&V  Team,  with  the  Air  Force  designated  as  lead  service. 
In  October  1983  the  Air  Force  officially  accepted  responsibility  as  lead  service  and  desig¬ 
nated  the  Air  Force  Wright  Aeronautical  Laboratories  (AFWAL)  at  Wright  Patterson  Air 
Force  Base  as  lead  organization.  In  April  1984  an  E&V  Workshop  was  held  at  Airlie,  Vir¬ 
ginia.  The  purpose  of  the  workshop  was  to  solicit  participation  of  industry  representatives 
in  the  E&V  Task.  Many  of  the  participants  in  the  workshop  have  chosen  to  remain  in¬ 
volved  as  Distinguished  Reviewers,  and  additional  industry  participants  have  subsequently 
become  involved  in  E&V  Team  activities. 

The  E&V  Task  publishes  an  annual  public  report.  The  following  paragraph  is 
quoted  from  the  1987  version  [@E&V  Report  1987]  of  the  report: 


“The  Ada  community,  including  government,  industry,  and  academic  person¬ 
nel,  needs  the  capability  to  assess  APSEs  (Ada  Programming  Support  Environ¬ 
ments)  and  components  and  to  determine  their  conformance  to  applicable 
standards  (e.g.,  DoD-STD-1838,  the  CAIS  standard).  The  technology  required 
to  fully  satisfy  this  need  is  extensive  and  largely  unavailable;  it  cannot  be  ac¬ 
quired  by  a  single  government-sponsored,  professional  society-sponsored,  or 
private  effort.  The  purpose  of  the  APSE  Evaluation  and  Validation  (E&V)  Task 
is  to  provide  a  focal  point  for  addressing  the  need  by  (1)  identifying  and  defin¬ 
ing  specific  technology  requirements,  (2)  developing  selected  elements  of  the 
required  technology,  (3)  encouraging  others  to  develop  some  elements,  and 
(4)  collecting  information  describing  existing  elements.  This  information  will  be 
made  available  to  DoD  components,  other  government  agencies,  industry,  and 
academia.” 


The  team  public  reports  contain  much  additional  information  for  the  interested  reader.  See 
for  example,  the  “DoD  APSE  Analysis  Report”  [@E&V  Report  1984],  the  "Requirements  for 
the  Evaluation  and  Validation  of  Ada  Programming  Support  Environments,  Version  2.0” 
[ra-E&V  Report  1987],  and  the  "Tools  and  Aids  Document,  Version  1.0”  [@E&V  Report 
1987],  which  are  synopsized  in  Chapter  4  [4.3,  4.5,  4.6]. 
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Three  competitive  contracts  have  been  awarded  under  the  E&V  Task.  These 
are: 

•  Technical  Support  contract  -  awarded  June  1985 

•  Ada  Compiler  Evaluation  Capability  (ACEC)  contract  -  awarded 
February  1987 

•  CAIS  Implementation  Validation  Capability  (CIVC)  contract  - 
awarded  May  1987. 

The  major  purpose  of  the  first  of  these  contracts  is  to  create  and  update  ele¬ 
ments  of  the  E&V  Reference  System,  including  this  document.  The  purpose  of  the  sec¬ 
ond  and  third  contracts  is  to  create  two  specific  elements  (ACEC  and  CIVC)  of  the 
needed  E&V  technology. 


1.4  ORGANIZATION  OF  THE  GUIDEBOOK 

Chapter  2  provides  a  general  description  of  the  structure  and  use  of  the 

Guidebook. 


Chapter  3  provides  high-level  guidance  to  users  who  may  need  assistance  in 
selecting  instances  of  the  technology  and  integrating  the  results  of  its  application. 

Chapter  4  provides  synopses  of  other  documents  or  activities  that  are  either 
too  broad  in  scope  to  fit  within  one  of  the  later  chapters  or  are  of  historical  importance  to 
E&V  activities. 

Chapter  5  and  subsequent  chapters  are  "formal  chapters”  that  describe  or  re¬ 
fer  to  specific  instances  of  E&V  technology.  Each  of  the  formal  chapters  contains  all  of 
the  procedures  and  techniques  associated  with  a  particular  group  of  tools  or  toolsets  to  be 
assessed,  such  as  Compilation  System  Assessors  or  Test  System  Assessors.  A  standard 
format  based  on  a  "formal  grammar"  is  used  in  presenting  this  material.  See  further  ex¬ 
planations  in  Appendix  C. 
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Appendices  A,  B,  and  C  contain  a  list  of  citations,  a  list  of  acronyms  and  ab¬ 
breviations,  and  a  definition  of  the  formal  grammar  used  in  the  formal  chapters,  respec¬ 
tively.  Appendix  D  contains  the  list  of  vendors  and  agents  of  assessment  tools  who  are 
the  primary  sources  of  E&V  technology. 
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2.  STRUCTURE  AND  USE  OF  THE  GUIDEBOOK 

This  chapter  provides  a  brief  explanation  of  the  structure  and  uses  of  the  E&V 
Guidebook.  It  is  expected  that  many  users  have  first  consulted  the  E&V  Reference  Manual 
(see  Fig.  1.1-1)  and  come  to  the  Guidebook  with  a  specific  chapter  and  section  number  in 
hand,  prepared  to  read  about  a  specific  instance  of  E&V  technology.  A  user  following  this 
path  does  not  particularly  care  about  the  overall  structure  of  the  document.  Other  users, 
however,  may  come  to  the  document  with  a  less  narrowly-defined  objective.  An  attempt 
has  been  made,  with  such  users  in  mind,  to  make  the  Guidebook  easy  to  use  as  a  stand¬ 
alone  document. 


2.1  STRUCTURE 

The  Guidebook  structure  may  be  considered  as  having  four  major  subdivisions, 

as  follows: 

•  Introductory  Material  (Chapters  1  and  2) 

•  General  Background  Material  (Chapters  3  and  4) 

•  Specific  E&V  Technology  Descriptions  (Chapters  5  and  beyond) 

•  Appendices. 

The  introductory  material  is  used  to  introduce  the  document  and  its  structure.  The  general 
background  material  is  used  to  introduce  the  general  subject  of  APSE  assessment.  Chap¬ 
ter  3  is  an  "essay”  designed  to  help  users  who  are  faced  with  the  question  of  how  to 
evaluate  an  APSE  as  a  whole,  or  how  to  compare  several  APSEs  with  the  objective  of  se¬ 
lecting  one.  (Chapter  3  of  the  E&V  Reference  Manual,  dealing  with  whole  APSE  assess¬ 
ment  issues,  is  a  "companion  essay”  that  provides  complementary  background  material.) 
Chapter  4  provides  a  different  kind  of  background  material.  It  may  be  considered  a  “guide 
to  the  literature"  of  APSE  assessment.  It  contains  synopses  of  documents  that  fall  into 
one  of  two  categories.  One  category  is  that  of  documents  that  contain  no  specific 
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instances  of  E&V  technology,  but  contain  generai'y  useful  background  material.  The  other 
category  is  that  of  documents  that  contain  or  discuss  multiple  instances  of  E&V  technol¬ 
ogy.  which  are  individually  covered  in  multiple  parts  of  the  later,  formal  chapters.  These 
multiple  instances  can  be  thought  of  as  children  of  a  common  parent.  In  order  to  avoid 
the  redundancy  of  summarizing  the  parent  document  many  times,  the  Chapter  4  synopsis 
is  provided  as  a  common  point  to  which  all  the  children  may  refer.  Each  “synopsis  text 
frame”  in  Chapter  4  has  the  following  parts: 

•  Citation:  (the  primary  reference) 

•  Synopsis:  (brief  description) 

•  Methods:  (references  to  specific  instances  of  E&V  technology,  if 
any). 

The  formal  chapters  (Chapters  5  and  beyond),  which  comprise  the  main  bulk 
of  the  Guidebook,  describe  or  summarize  specific  instances  of  E&V  technology.  The 
chapter  subjects  and  titles  were  chosen  to  be  meaningful  and  intuitive  to  users  of  the 
Guidebook.  Thus,  they  focus  on  the  subject  of  assessment  (e.g..  Compilation  System, 
Test  System,  Ada  Design  Support  System,  etc)  rather  than  the  method  of  assessment 
(e.g.,  formal  validation,  subjective  evaluation,  etc).  Within  each  chapter  there  are,  in  gen¬ 
eral,  multiple  instances  of  assessment  technology.  Some  may  be  examples  of  evaluation 
techniques,  others  may  be  examples  of  validations,  others  may  be  mixtures  of  the  two. 
Readers  should  not  infer  approval  of  the  E&V  Task,  because  a  tool  or  technique  is  in¬ 
cluded  in  the  collection,  or  disapproval,  because  a  tool  or  technique  is  not  included. 
Readers  who  know  of  instances  of  E&V  technology  not  reported  here  are  urged  to  contact 
the  E&V  Task  chairman,  in  the  manner  described  in  the  Executive  Summary.  The  sepa¬ 
rate  instances  within  a  chapter  are  simply  placed  there  in  chronological  order,  indicating 
the  relative  timing  of  the  material’s  first  appearance  in  the  Guidebook.  Readers  should  not 
infer  anv  judgment  as  to  relative  importance  based  on  order.  Each  chapter  thus  provides 
a  dynamically  growing  section  of  the  Guidebook.  Old  sections  will  not  be  thrown  away  or 
replaced  by  new  sections  describing  newer  techniques.  Old  sections  may,  however,  be 
updated  if  a  particular  vendor  or  agent  has  updated  material  describing  a  technique,  or 
has  improved  the  technique  itself  witnout  fundamentally  changing  the  approach.  Each 
"technique  text  frame”  in  the  formal  chapters  has  the  following  parts: 
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•  Purpose: 

•  Primary  References: 

•  Host/OS:  (if  applicable) 

•  Vendors/Agents:  (if  applicable) 

•  Method: 

Inputs: 

Process: 

Outputs: 


The  final  major  subdivision  (the  appendices)  require  little  explanation  here. 
The  formal  grammar  described  in  Appendix  C  need  not  concern  most  users.  It  was  em¬ 
ployed  oecause  of  the  possibility  of  a  future  on-line,  electronic  version  of  the  Reference 
System,  supported  by  advanced  updating  and  information  retrieval  techniques. 


2.2  EXAMPLE  USES 

Instances  of  E&V  technology  may  be  found  in  two  ways.  A  user  may  consult 
the  Guidebook  directly,  or  may  first  consult  the  E&V  Reference  Manual,  as  pictured  in  Fig¬ 
ure  1.1-1.  A  user  who  comes  directly  to  the  Guidebook  would  typically  first  look  at  the  Ta¬ 
ble  of  Contents.  For  example,  a  user  interested  in  evaluating  compiler  performance  would 
naturally  look  under  Chapter  5  “Compilation  System  Assessors."  The  titles  of  Sec¬ 
tions  5.2,  “IDA  Benchmarks,"  and  5.3,  “Ada  Compiler  Evaluation  Capability  (ACEC),” 
would  probably  suggest  themselves  as  relevant  to  this  user’s  needs  —  as  indeed  they  are. 

Alternatively,  the  user  may  consult  the  E&V  Reference  Manual,  which  is  de¬ 
signed  to  help  find  E&V  techniques  in  the  same  way  that  the  card  catalog  helps  people 
find  books  in  the  library.  For  example,  the  Reference  Manual  contains  both  a  Function  In¬ 
dex  and  an  Attribute  Index,  each  of  which  contains  cross  references  to  elements  in  the 
other.  One  element  of  the  Function  Index  is  the  function  "Compilation,”  which  is  cross-ref¬ 
erenced  to  a  number  of  relevant  attributes.  Under  the  particular  function-attribute  pair 
“Compilation-Processing  Effectiveness”  are  listed  a  number  of  Guidebook  references. 
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Among  these  are  the  same  two  Sections,  5.2  and  5.3,  of  the  Guidebook,  mentioned  in  the 
previous  paragraph.  The  user  following  this  procedure  could  pick  up  the  Guidebook  and 
go  directly  to  these  two  sections  or  “text  frames"  and  find  summary  information  concerning 
the  IDA  Benchmarks  test  suite  and  the  ACEC  test  suite,  respectively. 
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3.  INTEGRATION  OF  APSE  ASSESSMENTS 


The  purpose  of  this  chapter  is  to  provide  high-level  guidance  for  the  user  of 
the  E&V  Reference  System  (Reference  Manual  and  Guidebook)  who  is  interested  in  evalu¬ 
ating  an  APSE  as  a  whole,  or  in  comparing  several  APSEs  with  the  objective  of  selecting 
one.  While  the  “formal  chapters"  (beginning  with  Chapter  4  of  the  Reference  Manual  and 
Chapter  5  of  the  Guidebook)  provide  assistance  in  locating,  defining,  and  assessing  many 
individual  aspects  of  APSEs,  they  do  not  provide  an  overall  approach  to  weighting  and 
combining  the  results  of  such  assessments.  Section  3.1  briefly  discusses  some  relevant 
general  background  material.  Section  3.2  discusses  some  earlier,  partial  efforts  aimed  at 
an  integrated  approach.  Section  3.3  provides  some  additional  guidance  leading  to  a  com¬ 
prehensive,  integrated  approach. 

It  is  necessary,  first,  to  distinguish  the  subject  of  this  chapter  --  integrated 
whole-APSE  assessment  —  from  the  subject  of  Chapter  13  —  specific  “Whole-APSE  Asses¬ 
sors.”  The  integrated  form  of  whole-APSE  assessment  (Section  3.3)  involves  a  combining 
or  mixing  together  of  the  results  of  individual  assessment  steps  to  arrive  at  a  decision. 
These  individual  steps  may  be  oriented  toward  specific  functions  or  tools,  or  may  be  ori¬ 
ented  toward  a  “whole  APSE,”  in  relation  to  specific  attributes  or  the  APSE’S  performance 
in  a  specific  life-cycle  phase  or  activity.  Thus,  a  whole  APSE  assessor  (Chapter  13)  might 
be  used  to  evaluate  the  APSE’S  capability  to  support  a  project  team  during  one  major  ac¬ 
tivity.  such  as  preliminary  design.  The  results  of  such  an  assessment  would  become  one 
of  the  weighted  factors  of  an  integrating  process  leading  to  a  major  decision. 


3.1  GENERAL  BACKGROUND 

Chapter  4  of  this  Guidebook  contains  synopses  of  books,  articles,  and  docu¬ 
ments.  Some  of  these  have  historical  value  and  are  also  indirectly  relevant  to  the  topic  of 
an  integrated  approach  to  APSE  evaluation  because  they  provide  definitions  of  an  APSE  or 
highlight  issues  that  may  be  important  during  APSE  evaluations.  The  Stoneman  document 
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[<®DoD  1980]  defines  an  APSE  as  a  layered  system  and  includes  some  discussion  of 
evaluation  criteria.  The  Common  APSE  Interface  Set  (CAIS)  and  CAIS-A  definition  docu¬ 
ments  [fg'DoD  1986.  @DoD  1988]  describe  proposed  interface  requirements  for  interfaces 
that  exist  between  layers  of  an  APSE.  The  motivation  for  these  interface  requirements  is 
to  support  the  transportability  of  tools  and  project  data  bases  from  one  APSE  to  another. 
The  book  “Life  Cycle  Support  in  the  Ada  Environment”  by  McDermid  and  Ripken 
[faMcDermid  1984]  takes  a  top-down  approach  to  defining  a  “coherent  APSE,"  starting 
with  requirements  for  a  coherent  life-cycle  methodology;  see  synopsis  [4.10].  Several  pa¬ 
pers  in  an  IEEE  Tutorial  [rg/Wasserman  1981]  provide  relevant  observations  on  desirable 
characteristics  and  major  issues  for  Ada  support  environments;  see  synopses  [4.11.  4.12, 
4.13].  A  more  recent  survey  paper  “Characteristics  and  Functions  of  Software  Engineering 
Environments:  An  Overview"  [^Houghton  and  Wallace  1987]  provides  a  broad  discussion 
of  environments  and  the  state  of  the  art;  see  synopsis  [4.15].  Chapter  3  of  the  E&V  Ref¬ 
erence  Manual  [foRM:  Whole  APSE  Issues  3.]  presents  various  ways  of  viewing  an  APSE 
and  key  whole-APSE  attributes. 


3.2  EARLY  EFFORTS  AT  INTEGRATED  APSE  ASSESSMENT 


The  following  quotation  is  from  a  paper  by  Henderson  and  Notkin  [r&Henderson 

1987]: 

“Perhaps  the  biggest  failing  of  environments  research  and  development 
to  date  is  the  general  lack  of  scientific  evaluation  of  existing  environ¬ 
ments.  Evaluation  approaches  and  actual  evaluations  are  beginning  to 
appear,  but  relatively  little  effort  has  been  given  to  this  undeniably  fun¬ 
damental  subject." 

Some  early  efforts  are  mentioned  briefly  below. 


The  Software  Engineering  Institute  (SEI)  has  developed  a  methodology 
[rS/Weiderman  1987]  to  evaluate  certain  aspects  of  APSEs.  The  methodology  centers 
around  the  execution  of  several  experiments  in  the  environment(s)  to  be  evaluated.  The 
experiments  are  designed  in  a  generic  fashion  and  must  be  tailored  or  "instantiated"  for 
each  specific  environment;  see  synopsis  [4.13], 
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The  book  “Selecting  an  Ada  Environment”  [@Lyons  1986],  written  by  the  Ada 
Europe  Environment  Working  Group,  provides  background  discussion  about  a  broad  range 
of  topics.  In  each  chapter  and  section  it  provides  a  list  of  questions  to  be  asked  about  the 
environment  under  consideration;  see  synopsis  [4.9]. 

It  is  apparent  that  industry  has  devoted  resources  internally  to  comparative  as¬ 
sessment  of  commercial  APSEs.  However,  little  has  yet  been  published  in  the  open  litera¬ 
ture  describing  the  techniques  employed. 


3.3  TOWARDS  A  COMPREHENSIVE  APPROACH 

The  published  literature  on  assessment  of  software  engineering  environments 
does  not  include  descriptions  of  “decision  support”  oriented  approaches.  A  decision  sup¬ 
port  system  is  one  that  leads  a  user  through  a  structured  framework  that  includes  weight¬ 
ing  factors  and  decision  criteria,  and  supports  a  final  decision  process.  As  applied  to 
APSE  assessment  this  kind  of  approach  would  support  a  final  decision,  such  as,  whether  a 
single  APSE  under  consideration  is  “good  enough,”  or  which  of  several  APSEs  under  con¬ 
sideration  is  'best.” 


The  following  characteristics  appear  to  be  appropriate  for  a  decision  support 
system  designed  for  integrated  APSE  assessment; 


•  The  system  should  allow  the  specification  of  a  list  of  "essential 
features"  that  are  absolutely  required  for  the  contemplated  applica¬ 
tion  or  family  of  applications.  Ideally,  each  of  these  essential  fea¬ 
tures  would  be  subject  to  a  question  or  test  that  yields  an 
unambiguous  "yes/no"  result. 

•  The  system  should  allow  the  specification  of  a  second  list  of  attrib¬ 
utes  and  function-attribute  pairs  that  represent  desirable  features  or 
criteria,  which  should  be  involved  in  an  integrated  assessment. 

•  The  system  should  allow  for  specification  of  "weights”  to  be  ap¬ 
plied  to  each  attribute  and  function-attribute  pair  in  the  second  list. 
The  weights  will  typically  be  chosen  subjectively  by  the  assess¬ 
ment  participmts. 
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•  The  system  should  include  a  mechanism  to  document/identify  the 
method  of  assessment  used  for  every  test/metric  to  be  employed 
in  addressing  every  essential  and  desirable  feature. 

•  The  system  should  include  a  well-defined  method  of  combination, 
leading  to  an  overall  set  of  pre-decision  results.  For  example,  the 
results  may  be  summarized  in  two  lines  as  in: 

1)  satisfies  all  essential  requirements  (listed  in  Table  A) 

2)  scores  72  out  of  possible  100  (based  on  weights  in  Table  B). 

The  characteristics  outlined  above  represent  a  general  framework  that  can  be  applied  very 
differently  by  different  users.  At  one  extreme  is  a  decision  maker  with  little  time  or  re¬ 
sources,  who  focuses  on  a  short  list  of  essential  features  only,  and  accepts  answers  sup¬ 
plied  by  vendors  or  vendor  documentation.  At  the  other  extreme  is  a  team  of  APSE 
assessors  who  conduct  a  comprehensive,  detailed  set  of  tests  and  “model  project”  experi¬ 
ments  and  expend  multiple  person-years  of  effort  in  a  comparative,  hands-on  assessment 
of  competing  APSEs. 

• 

It  is  also  possible  that  two  assessment  teams  applying  equal  resources  might 
differ  greatly  in  the  manner  of  their  assessments.  One  might  view  the  APSE  as  a  support 
system  for  a  particular  life-cycle  methodology  adopted  by  its  organization.  Another  might 
view  the  APSE  as  a  project  data  base  management  system.  These  two  teams  would  be 
likely  to  use  very  different  tests,  or  very  different  weights  where  the  same  tests  are  used. 
Neither  is  necessarily  right  or  wrong.  In  the  final  analysis,  it  is  the  software  developer’s  re¬ 
sponsibility  to  understand  his  own  application  area  and  the  most  critical  attributes  of  his  de¬ 
velopment  support  environment. 
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4. 


SYNOPSES 


The  purpose  of  this  chapter  is  to  provide  a  single  place  in  the  Guidebook  for 
synopses  of  documents  (or  other  resources),  which  have  too  broad  a  scope  to  fit  within 
one  of  the  subsequent  Chapters.  In  some  cases  the  subject  document  appears  only  in 
this  Chapter  because  it  does  not  contain  specific  instances  of  E&V  technology.  For  exam¬ 
ple,  the  Stoneman  document  [@DoD  1980]  does  not  deal  with  evaluation  or  validation  of 
APSEs,  but  it  has  general  historical  importance  to  the  entire  field  of  Ada  environments  and 
has  been  selected  as  the  first  document  to  be  synopsized.  In  other  cases  a  particular 
document  may  contain  multiple  instances  of  E&V  technology,  which  are  themselves  sum¬ 
marized  or  referenced  in  multiple  parts  of  the  Guidebook.  These  multiple  instances  can  be 
thought  of  as  children  of  a  common  parent.  In  order  to  avoid  the  redundancy  of  summa¬ 
rizing  the  parent  document  many  times,  the  Chapter  4  synopsis  is  provided  as  a  common 
point  to  which  all  the  children  may  refer.  The  formal  grammar  used  to  structure  the  entries 
in  subsequent  chapters  includes,  therefore,  a  mechanism  for  referring  back  to  the  synopsis 
contained  in  Chapter  4.  Similarly,  after  each  synopsis  there  is  a  provision  for  forward  ref¬ 
erences  to  specific  techniques  (if  any)  described  in  later  chapters. 

Most  of  the  documents  synopsized  in  this  chapter  are  readily  available  through 
public  sources.  A  few  of  them  may  be  difficult  or  impossible  to  obtain  for  some  readers; 
these  were  included  because  the  synopsis  itself  was  judged  to  be  helpful  in  filling  in  a 
piece  of  the  historical  background. 
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4.1  STONEMAN 


Citations: 

[ra>DoD  1980]  “Requirements  for  Ada  Programming  Support 
Environments  —  STONEMAN,"  1980. 

Synopsis: 

The  Stoneman  document  defines  the  APSE  as  a  layered  system.  The  innermost 
layer  is  referred  to  as  the  Kernel  APSE,  or  KAPSE.  The  KAPSE  is  machine- 
dependent  and  includes  the  database  functions  and  other  general  operating  system 
support  functions.  The  next  layer,  the  Minimal  APSE,  or  MAPSE,  consists  of  the 
minimal  set  of  tools  which  can  support  the  development  of  software.  The  outer¬ 
most  layer,  the  APSE,  consists  of  tools  and  functions  that  are  project  dependent. 

In  addition  to  providing  guidance  for  APSE  designers,  the  Stoneman  document  pro¬ 
vides  some  evaluation  criteria  for  APSEs. 
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4.2 


HOUGHTON:  A  TAXONOMY  OF  TOOL  FEATURES  FOR  THE  Ada 
PROGRAMMING  SUPPORT  ENVIRONMENT  (APSE) 


Citations: 

[Houghton  1983]  B.C.,  Houghton,  Jr.,  “A  Taxonomy  of  Tool  Features  for  the 
Ada  Programming  Support  Environment  (APSE),"  National  Bureau  of  Standards, 
NBSIR-81-2625,  February  1983. 

Synopsis: 

This  paper  puts  forth  a  taxonomic  classification  of  APSE  features.  The  features 
included  satisfy  the  criteria  that  they  are  “within  current  technology"  and  are  “ori¬ 
ented  to  the  Ada  language.”  The  top  two  levels  of  the  classification  are  as  fol¬ 
lows: 

Input 

Subject 
Control  Input 

Function 

Transformation 
Management 
Static  Analysis 
Dynamic  Analysis 

Output 

User  Output 
Machine  Output 

For  each  of  the  second-level  elements  above,  a  third-level  list  is  given,  and  some 
discussion  is  provided.  The  paper  includes  the  results  of  a  survey  in  which  the 
second  and  third-level  elements  under  “Function”  are  each  rated  as  “Required," 
“Important,”  or  “Useful.” 
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4.3  E&V  REPORT:  DoD  APSE  ANALYSIS 


Citations: 

[E&V  Report  1984]  “DoD  APSE  Analysis  Report,  Draft  Version  1.0,”  31  August  1984, 
Appendix  C  of  “Evaluation  and  Validation  (E&V)  Team  Public  Report”.  Air  Force 
Wright  Aeronautical  Laboratories,  November  1984. 

Synopsis: 

The  DoD  Ada  Programming  Support  Environment  (APSE)  Analysis  Document  was 
prepared  by  the  APSE  Working  Group  (APSEWG)  of  the  E&V  Team.  It  contains  a 
description  and  analysis  of  the  Ada  programming  support  environments  developed 
by  each  of  the  armed  services.  The  three  environments  analyzed  were  the  Air 
Force’s  Ada  Integrated  Environment  (AIE),  the  Army’s  Ada  Language  System 
(ALS),  and  the  Navy’s  Ada  Language  System/Navy  (ALS/N).  The  design  docu¬ 
mentation  was  used  to  determine  the  functionality  contained  in  each  programming 
environment.  The  functions  were  described  in  a  taxonomy  in  order  to  determine 
the  commonality  and  differences  of  each  system.  The  taxonomy  developed  for 
this  purpose  was  an  expanded  version  of  the  function  part  of  the  taxonomy  devel¬ 
oped  earlier  by  Houghton  [@Houghton  1983]:  see  synopsis  [4.2]. 
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4.4  CLASSIFICATION  SCHEMA/E&V  TAXONOMY  CHECKLISTS 

Citations: 

[E&V  Schema  1987]  “E&V  Classification  Schema  Report,”  TASC,  TR-5234-2, 
Version  1.0,  15  June  1987. 

Synopsis: 

The  purpose  of  this  document  was  to  set  forth  a  schema,  or  a  framework,  to  be 
used  in  subsequent  E&V  documents,  especially  the  E&V  Reference  Manual  [RM], 
The  Function  Index  of  the  schema  was  strongly  influenced  by  earlier  documents, 
such  as  Houghton's  taxonomy  [4.2],  the  DoD  APSE  Analysis  Report  [4.3],  and  the 
SEE  tool  features  taxonomy  [@Kean  1985].  The  upper  levels  of  the  Function  In¬ 
dex  of  the  schema  became  the  initial  version  of  the  Function  Index  of  the  Refer¬ 
ence  Manual.  The  lower  levels  were  found  to  incorporate  a  large  number  of  tool 
functions  which  could  be  evaluative  in  nature.  These  tool  function  features  have 
been  carried  over  into  the  Guidebook  as  capability  assessment  checklists.  As  a 
group,  they  are  considered  the  Classification  Schema  Checklists. 


Methods: 

[Compilation  Checklist  5.8; 

Program  Library  Management  Checklist  5.9; 

Linking/Loading  Checklist  6.2; 

Import/Export  Capabilities  Checklist  6.3; 

Emulation  Capabilities  Checklist  6.4; 

Debugging  Capabilities  Checklist  6.5; 

Tuning  Analysis  Capabilities  Checklist  6.7; 

Real-Time  Analysis  Capabilities  Checklist  6.8; 

Configuration  Management  Capabilities  Checklist  10.1; 
Text  Editing  Capabilities  Checklist  99.1; 

Electronic  Mail  Capabilities  Checklist  99.3] 
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4  5  REQUIREMENTS  FOR  E&V 


Citations: 

[@E&V  Report  1987]  “Requirements  for  Evaluation  and  Validation  of  Ada  Program¬ 
ming  Support  Environments,  Version  2.0.”  4  December  1986.  Appendix  D  of 
“Evaluation  and  Validation  (E&V)  Team  Public  Report,”  Air  Force  Wright 
Aeronautical  Laboratories,  September  1987. 

Synopsis: 

This  document  was  prepared  by  the  Requirements  Working  Group  (REQWG)  of  the 
E&V  Team.  Its  purpose  is  to  set  forth  requirements  on  the  E&V  Task.  It  is  in¬ 
tended  for  use  by  the  E&V  Team  and  by  the  E&V  Task  contractors  in  developing 
technology  for  the  evaluation  and  validation  of  APSEs.  However,  its  use  in  other 
E&V  efforts  is  encouraged.  The  document  contains  three  categories  of  require¬ 
ments:  (i,  those  on  the  E&V  Team  itself,  (2)  those  on  the  E&V  methods  and  pro¬ 
cedures,  and  (3)  those  specifying  what  is  to  be  evaluated  or  validated.  See  also 
the  Tools  and  Aids  Document,  synopsis  [4.6]. 
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4.6  TOOLS  AND  AIDS  FOR  E&V 

Citations: 

[@E&V  Report  1987]  “Tools  and  Aids  Document,  Version  1.0,”  September  1987, 
Appendix  C  of  “Evaluation  and  Validation  (E&V)  Team  Public  Report,”  Air  Force 
Wright  Aeronautical  Laboratories,  September  1987, 

Synopsis: 

This  document  was  prepared  by  the  Requirements  Working  Group  (REQWG)  of  the 
E&V  Team,  It  identifies  the  community’s  E&V  technology  needs,  provides  defini¬ 
tions  of  those  needs,  and  prioritizes  them  in  order  of  their  relative  importance.  The 
purpose  of  this  document  is  to  provide  pertinent  information  to  those  agencies  will¬ 
ing  and  able  to  fund  the  development  of  E&V  technology.  It  reflects  the  E&V  Re¬ 
quirements  Document  (see  synopsis  [4.5])  and  views  on  the  subject  obtained  from 
surveys  conducted  among  the  E&V  Team  and  appropriate  ARPANET-MILNet  Inter¬ 
est  Groups. 
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4.7  STARS-SEE  OPERATIONAL  CONCEPT  DOCUMENT 


Citations: 

[STARS-SEE  1985]  “Proposed  Version  001.0,”  STARS  Joint  Service  Team  for 
Software  Engineering  Environments,  Stars  Joint  Program  Office,  October  1985. 

Synopsis: 

The  Software  Technology  for  Adaptable,  Reliable  Systems  —  Software  Engineering 
Environment  (STARS-SEE)  Operational  Concept  Document  (OCD)  presents  require¬ 
ments  from  the  perspective  of  the  STARS-SEE  users.  It  represents  a  consensus 
among  the  Government  agencies  responsible  for  SEE  development  and  support, 
STARS-SEE  implementors,  and  potential  users.  Major  sections  of  the  document 
describe  the  STARS-SEE  mission,  operational  and  support  environments,  and  sys¬ 
tem  components  and  functions.  The  primary  mission  centers  on  the  development, 
support,  reuse,  management,  and  control  of  mission  critical  software.  The  STARS- 
SEE  system  is  defined  to  consist  of  the  people,  computers,  software,  and  proce¬ 
dures  needed  to  perform  the  mission.  Major  topics  discussed  include  (1)  the  types 
of  users  and  associated  software  activities.  (2)  the  function  of  the  Integration  and 
Compatibility  Framework,  (3)  the  capabilities  required  by  the  Information  Storage 
and  Retrieval  System,  (4)  the  functional  capabilities  of  the  SEE,  (5)  the  SEE-user 
interaction,  and  (6)  the  hardware  and  software  characteristics  of  the  computer  sys¬ 
tem.  The  functional  capabilities  address  project  planning  and  control,  requirements 
specification  and  analysis,  design  specification  and  analysis,  software  prototyping 
and  modeling,  reusability,  program  generation  and  unit  testing,  integration  testing, 
quality  assurance,  verification  and  validation,  configuration  management,  software/ 
hardware  integration,  post  deployment  software  support,  project  communications, 
generation  of  documents,  data  collection,  performance  and  productivity  measure¬ 
ment,  help  and  training  for  STARS-SEE  users,  the  transition  to  and  tailoring  of  the 
STARS-SEE,  and  knowledge  engineering. 
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4.8  GRUND,  ET  AL.:  KEY  CHARACTERISTICS  OF  APSES 


Citations: 

[Grund  1985]  E.C.  Grund,  L.A.  Hilliard,  and  K.A.  Younger,  “Key  Characteristics 
of  Ada  Programming  Support  Environments,"  MITRE  Corporation,  ESD  — 

TR-85-144,  MTR-9590,  July  1985. 

Synopsis: 

This  document  is  intended  to  provide  basic  information  about  Ada  Programming 
Support  Environments  for  people  concerned  with  the  specification  or  selection  of 
an  APSE.  Section  1  summarizes  the  STONEMAN  APSE  requirements.  Section  2 
describes  desirable  characteristics  of  APSEs  in  five  areas:  compilers,  run-time  en¬ 
vironments,  databases,  configuration  management  tools,  and  editors.  A  short  list 
of  questions  to  ask  in  each  area  is  included.  Section  3  describes  four  Ada  pro¬ 
gramming  support  products  available  or  under  development  in  early  1985  in  terms 
of  their  capabilities  in  the  same  five  areas. 
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4.9  Ada-EUROPE:  SELECTING  AN  Ada  ENVIRONMENT 


Citations: 

[Lyons  1986]  “Selecting  an  Ada  Environment."  eds.  T.G.L.  Lyons  and  J.C.D. 

Nissen,  Ada-Europe  Working  Group,  Cambridge  University  Press,  1986. 

Synopsis: 

The  Ada-Europe  Environment  Working  Group,  under  the  chairmanship  of  John 
Nissen,  produced  a  guide  which  adopts  the  “point  of  view  of  a  potential  user  wish¬ 
ing  to  select  an  environment,  and  provides  lists  of  questions  to  be  asked  about  the 
environment  under  consideration."  It  generally  follows  the  structure  proposed  in 
Stoneman  [@DoD  1980]:  it  “starts  from  the  inside  of  the  onion  structure  and  works 
outwards.”  Each  of  its  19  Chapters  follows  a  standard  format.  Topics  are  intro¬ 
duced  and  discussed,  typically  using  one  or  two  pages  of  text,  and  then  a  list  of 
appropriate  questions  is  provided. 
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4.10  MCDERMID  AND  RIPKEN:  LIFE  CYCLE  SUPPORT  IN  THE  Ada  ENVIRONMENT 


Citations: 

[McDermid  1984]  J.  McDermid  and  K.  Ripken,  “Life  Cycle  Support 
in  the  Ada  Environment,"  Cambridge  University  Press,  1984. 

Synopsis: 

This  book  contrasts  its  own  approach  to  APSE  development  with  that  of  the 
Stoneman  report  [@DoD  1980].  Stoneman  takes  a  bottom-up  approach,  starting 
with  a  kernel  and  minimal  APSE  (KAPSE  and  MAPSE),  as  a  foundation  for  exten¬ 
sions  to  more  powerful  and  better  integrated  environments.  McDermid  and  Ripken 
follow  a  top-down  approach  by  defining  requirements  for  a  coherent  life-cycle 
methodology.  They  then  describe  a  particular  instance  of  a  coherent  methodol¬ 
ogy,  as  a  combination  of  existing  methods  used  in  various  life-cycle  phases.  This 
description  becomes  the  basis  for  a  definition  of  a  “coherent  APSE"  that  supports 
the  entire  life  cycle. 

The  authors  use  a  seven-phase  life  cycle  and  state  requirements  for  each  phase  in 
terms  of  (1)  a  system  representation  form,  (2)  a  transformation  method  and  (3)  a 
verification  activity.  Table  4.8-1  lists  the  names  of  the  seven  phases  (each  named 
for  its  principal  output)  and  the  methods  selected  for  each. 

TABLE  4.8-1 

EXAMPLE  COHERENT  METHODOLOGY 


PHASE  (OUTPUT) 

SELECTED  METHOD 

Requirements  Expression 

CORE 

System  Specification 

A-7  Techniques 

Abstract  Functional  Specification 

A-7  Techniques 

Module  Specification 

Ada  and  ANNA 

Module  Design 

Ada  and  ANNA 

Module  Code 

Ada  and  ANNA 

Executable  System 

- 
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The  authors  are  not  completely  satisfied  with  all  of  the  methods  chosen,  and 
point  out  shortcomings  in  each  case.  They  suggest  the  book  be  used  as  “a  refer¬ 
ence  point  for  further  work  on  APSE  design  and  development.”  They  stress  that 
the  coherence  of  the  methods  and  ease  of  transition  from  one  phase  to  the  next  is 
an  important  attribute.  They  also  outline  a  phased  development  plan  in  which  a 
larger  scale  APSE  might  be  developed  in  the  following  three  steps:  (1)  a  “Clerical 
Support  APSE,”  (2)  a  “V&V  and  Management  Support  APSE,”  and  (3)  a  “Transfor¬ 
mation  Support  APSE.” 
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4.11  NOTKIN  AND  HABERMANN:  SOFTWARE  DEVELOPMENT  ENVIRONMENT 
ISSUES  AS  RELATED  TO  Ada 


Citalions: 

[Notkin  1981]  D.S.  Notkin  and  A.N.  Habermann,  “Software  Development 
Environment  Issues  as  Related  to  Ada,”  in  “Tutorial:  Software  Development 
Environments,”  ed.  A.I.  Wasserman,  IEEE,  1981,  pp.  107-133. 

Synopsis: 

This  paper  addresses  software  development  problems  that  arise  in  three  areas: 
programming,  system  composition,  and  management.  In  each  area  traditional 
methods  and  tools  are  contrasted  with  a  more  integrated  approach  exemplified  by 
an  experimental  environment  named  Gandalf. 

“Programming  issues  are  those  that  arise  when  a  single  programmer  takes  a 
program  all  the  way  from  its  specifications  to  a  working  program.” 

'  System  composition  issues  are  those  thai  arise  when  a  system  (or  a  version  of  a 
system)  is  built  by  integrating  many  programs  into  one.”  “The  two  basic  problems 
in  system  composition  are  interface  control  and  version  control.”  Traditional  meth¬ 
ods  use  isolated  tools  “coordinated  by  memory... or  scraps  of  paper.” 

“Management  issues  are  those  that  arise  when  a  group  of  more  than  one  person 
develops  and  maintains  a  system  over  a  period  of  time.”  Three  problem  catego¬ 
ries  are  addressed:  misunderstanding,  lack  of  information,  and  conflict  of  interest. 
Traditionally,  these  problems  have  been  handled  by  non-technical  means.  The 
problem  with  the  management  approach  to  a  management  environment  is  that  the 
solution  to  human  interaction  difficulties  is  treated  by  the  introduction  of  more  hu¬ 
man  interaction. 

Methods: 

Although  this  paper  was  not  written  as  an  example  of  E&V  technology,  the  following 
list  of  environment  software  requirements  (paraphrased  from  the  paper)  may  be 
used  as  a  high-level  checklist: 

•  Concurrent  multiple  users  must  be  supported 

•  An  efficient  implementation  of  Ada  must  be  possible 

•  Efficient  support  for  data  base  manipulations  is  needed 

•  A  good  file  system  is  essential 

•  An  extensible  command  language  is  needed. 

It  is  also  pointed  out  that  the  most  important  hardware  requirement  is  that  the 
software  requirements  listed  above  must  be  supported. 


4-13 


E&V  Guidebook,  Version  1.1 


4.12  STENNING,  ET  AL.:  THE  Ada  ENVIRONMENT:  A  PERSPECTIVE 


Citations: 

[Stenning  1981]  V.  Stenning,  T.  Froggart,  R.  Gilbert,  and  E.  Thomas,  “The  Ada 
Environment:  a  Perspective,"  in  "Tutorial:  Software  Development  Environments,” 
ed.  A.I.  Wasserman,  IEEE,  1981,  pp.  36-46. 

Synopsis: 

This  paper  discusses  the  objectives  and  the  design  of  the  Ada  Programming 
Support  Environment.  It  is  strongly  influenced  by  the  United  Kingdom  Ministry 
of  Defense  Ada  Support  System  Study,  which  was  initiated  by  the  MoD  in  January 
1979.  According  to  the  paper,  the  DoD  KAPSE/MAPSE/APSE  approach  is 
strongly  recommended  to  achieve  portability.  The  APSE  should  be  designed  to 
support  a  project  throughout  its  life  cycle.  Furthermore,  it  should  be  an  open- 
ended  environment.  This  would  allow  for  the  user  to  extend  or  modify  existing 
tools.  A  basic  configuration  control  manager,  a  complete  user  interface,  and  a 
complete  basic  tool  set  are  necessary  to  develop  an  Ada  Environment  which  will 
improve  program  reliability,  life-cycle  program  costs,  and  promote  portability. 
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4.13  WEIDERMAN:  EVALUATION  OF  Ada  ENVIRONMENTS 


Citations: 

[Weiderman  1987]  N.  Weiderman  and  N.  Haberman,  “Evaluation  of  Ada  Environments, 
Software  Engineering  Institute,  Technical  Report  CMU/SEI-87-TR-1,  March  1987. 

Synopsis: 

In  response  to  the  lack  of  available  research  about  the  selection  of  APSEs,  the 
Software  Engineering  Institute  (SEI)  has  developed  a  methodology  to  evaluate 
these  environments.  The  methodology  centers  around  the  execution  of  several 
experiments  in  the  environment  to  be  evaluated.  Several  experiments  have  been 
developed  in  the  following  areas:  System  Management;  Configuration  Manage¬ 
ment/Version  Control;  Design  and  Code  Development;  Unit  Testing  and  Debugging. 
The  environments  are  evaluated  in  terms  of  functionality,  performance,  user  inter¬ 
faces.  and  system  interfaces.  The  need  for  an  evaluator  to  tailor  an  evaluation 
technique  to  a  specific  environment  is  addressed  by  the  SEI  study.  The  experi¬ 
ments  that  have  been  designed  are  generic  experiments.  The  evaluator  derives, 
or  “instantiates,1’  the  environment-specific  technique  from  the  generic  experiment. 

In  the  final  phase  of  the  evaluation,  the  results  are  analyzed.  An  advantage  of  the 
application  of  this  methodology  is  that  results  can  be  compared  from  one  environ¬ 
ment  to  another.  See  also  a  paper  describing  an  application  of  the  SEI’s  method 
[fa)Gray  1987], 

Methods: 

[SEI  Unit  Testing  and  Debugging  Experiment  7.2; 

SEI  Design  Support  Experiment  9.1; 

SEI  Configuration  Management  Experiment  10.2] 
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4.14  BARSTOW  AND  SHROBE:  OBSERVATIONS  ON  INTERACTIVE 
PROGRAMMING  ENVIRONMENTS 

Citations: 

[Barstow  1981]  O.R.  Barstow  and  H.E.  Shrobe,  “Observations  on  Interactive 
Programming  Environments,"  in  “Tutorir1:  Software  Development  Environments,” 
ed.  A.I.  Wasserman,  IEEE,  1981,  pp.  285-301. 

Synopsis: 

This  paper  reviews  key  features  of  LISP-based  environments  and  comments  upon 
lessons  learned  from  these  environments  and  future  directions.  These  environ¬ 
ments  encourage  a  “progressive  enrichment”  style  of  development  rather  than  de¬ 
velopments  broken  into  distinct  phases  such  as  specification,  implementation,  and 
maintenance.  The  following  set  of  lessons  (described  more  fully  in  the  paper)  are 
concerned  with  the  programmer's  perception  of  the  environment: 

•  It  ;s  important  to  be  able  to  run  an  incomplete  program. 

•  The  user  should  be  able  to  view  the  program  from  many  different 
natural  viewpoints,  most  of  which  are  “structured”  in  nature. 

•  Intercommunication  among  tools  is  extremely  important. 

•  The  programmer  should  not  be  required  to  know  the  details  of  the 
particular  language  definition  used  in  the  current  implementation. 

•  The  environment's  interface  must  be  highly  tuned  to  be  as  natural 
as  possible  for  the  human  programmer. 

Environment  characteristics  created  with  these  lessons  in  mind  “lead  to  the  ultimate 
goal  of  a  programming  environment  (which  is  to  increase  the  ability  of  the  pro¬ 
grammer  to  communicate  with  the  computer)  by  taking  advantage  of  as  many 
naturally  occurring  structures  as  possible.” 
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4.15  HOUGHTON  AND  WALLACE:  CHARACTERISTICS  AND  FUNCTIONS 
OF  SOFTWARE  ENGINEERING  ENVIRONMENTS:  AN  OVERVIEW 


Citations: 

[Houghton  and  Wallace  1987]  R.C..  Houghton,  Jr.  and  D.R.,  Wallace,  “Charac¬ 
teristics  and  Functions  of  Software  Engineering  Environments:  An  Overview,” 

ACM  Software  Engineering  Notes,  Vol.  12  Number  1,  January  1987. 

Synopsis: 

This  paper  provides  a  comprehensive  discussion  of  software  engineering  environ¬ 
ments  in  general,  with  no  focus  on  Ada  or  any  specific  language.  Some  major 
topics  discussea  are: 

•  Environment  Types  and  Life  Cycle  Relationships 

•  Integration 

•  Human  Factors 

•  Analysis  and  Software  Quality 

•  Support  for  Different  Types  of  Users 

•  Support  for  Application 

•  Hardware  Support 

•  Levels  of  Support. 

In  its  concluding  section,  the  paper  stresses  that  software  engineering  environments 
should  be  viewed  as  systems  that  support  broad  categories  of  users  and  tasks 
throughout  the  full  life  cycle. 
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4.16  CAIS 


Citations: 

[DoD  1986]  DoD-STD-1838,  Common  APSE  Interface  Set  (CAIS), 

U.S.  Department  of  Defense,  9  October  1986. 

Synopsis: 

DoD-STD-1838,  hereafter  called  CAIS,  was  developed  by  the  KAPSE  Interface 
Team  (KIT)  and  the  KAPSE  Interface  Team  for  Industry  and  Academia  (KITIA)  dur¬ 
ing  the  period  from  1981  to  1986  as  a  first  evolutionary  step  towards  a  full  state-of- 
the-art  common  APSE  interface  standard. 

The  CAIS  is  designed  to  promote  source-level  portability  of  Ada  programs, 
especially  Ada  software  development  tools.  The  goal  of  the  CAIS  is  to  promote  in¬ 
teroperability  (of  database  objects)  and  transportability  (of  APSE  tools)  of  Ada  soft¬ 
ware  across  Department  of  Defense  (DoD)  APSEs.  The  scope  of  the  CAIS 
includes  interfaces  to  those  services,  traditionally  provided  by  an  operating  system, 
that  affect  tool  transportability.  The  CAIS  contains  definitions  for  a  set  of  Ada 
package  specifications  that  will  provide  a  standard  and  transportable  set  of  inter¬ 
faces  to  a  collection  of  such  underlying  services.  See  also  [@DoD  1988]  and  syn¬ 
opsis  [4.17],  and  the  overview  paper  [@Oberndorf  1988]. 
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4.17  CAIS-A 


Citations: 

[DoD  1988]  “Common  APSE  Interface  Set,  Revision  A,"  Proposed  DoD-STD-1838A, 
20  May  1988. 

Synopsis: 

CAIS-A  is  a  set  of  Ada  package  interfaces  designed  to  enhance  the  transportability 
of  Ada  Support  Environment  Tools.  The  scope  of  the  CAIS-A  includes  the  func¬ 
tionality  affecting  transportability  that  is  needed  by  tools,  but  not  provided  by  the 
language.  The  proposed  CAIS-A  contains  definitions  for  an  entity  management 
system  for  software  engineering  tools.  The  primitive  entities  defined  allow  for  the 
manipulation  of  devices,  files  and  processes.  CAIS-A  is  based  on  an  entity-rela¬ 
tionship  approach  and  it  allows  the  user  to  define  entities,  in  a  limited  way,  by 
means  of  a  typing  mechanism.  CAIS-A  also  includes  functionality  to  support  tools 
requiring  transaction  processing,  a  rudimentary  triggering  mechanism,  and  explicit 
control  over  APSE  distribution. 

The  CAIS-A  was  developed  by  SofTech  under  contract  to  Naval  Ocean  Systems 
Center.  CAIS-A  is  a  design  enhancement  to  the  existing  CAIS  (DoD-STD-1838) 
[@DoD  1986]:  see  synopsis  [4.16],  which  was  developed  by  the  KIT  and  KITIA  as 
a  first  evolutionary  step  towards  a  full,  state-of-the-art  interface  standard.  Design¬ 
ers  view  CAIS-A  as  the  next  step  in  that  evolutionary  process. 
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4.18  HOGAN  AND  PRUD’HOMME:  DEFINITION  OF  A 
PRODUCTION  QUALITY  COMPILER 


Citations: 

[Hogan  1985]  M.O.  Hogan,  and  S.M.  Prud’homme,  "Definition  of  a  Production 
Quality  Compiler,”  Aerospace  Corporation,  Technical  Report,  July  1985. 

Synopsis: 

The  study  that  led  to  this  report  was  sponsored  by  the  Space  Division  of  the  Air 
Force  Systems  Command.  The  report  “is  organized  as  a  set  of  prototype  require¬ 
ments,  along  with  guidance  on  how  to  tailor  the  requirement  for  specific  application 
areas.  In  this  form  it  can  be  used  either  as  a  tool  to  help  determine  whether  a 
particular  compiler  is  of  production  quality  or  as  a  guide  for  preparing  requirements 
for  compilers  to  be  used  in  the  development  and  maintenance  of  software  for  mis¬ 
sion  critical  computer  resources.” 

Chapters  2  through  6  address  interface  requirements:  user  interfaces,  machine 
interfaces,  runtime  system  interfaces,  compiler  related  components  interfaces,  and 
Ada  language  interfaces,  respectively.  Chapter  7  addresses  capacity  and  perform¬ 
ance  requirements,  Chapter  8  addresses  reliability  requirements.  Chapter  9  ad¬ 
dresses  documentation  requirements.  Each  of  the  above  chapters  follows  a 
standard  format  in  which  a  requirement  is  stated  in  the  form:  “The  compiler  shall 
.  .  .  ,"  and  then  a  “Guidance”  section  is  provided  giving  background  information 
and  help  in  subsetting  or  tailoring  the  requirement  for  specific  application  domains. 
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4.19  NISSEN,  ET  AL:  GUIDELINES  FOR  Ada  COMPILER 
SPECIFICATION  AND  SELECTION 

Citations: 

[Nissen  1984]  J.C.D.  Nissen,  B.A.  Wichman,  et  al.. 

“Guidelines  for  Ada  compiler  specification  and  selection,”  in  Ada:  Language, 
compilers  and  bibliography ,  ed.  M.W. Rogers,  Cambridge  University  Press,  1984. 

Synopsis: 

Members  of  Ada-Europe  produced  this  set  of  guidelines  based  upon  a  taxonomy  of 
compiler  features.  Their  caveat  is  clear:  “The  relative  value  of  information  about 
different  features  of  the  compiler  is  a  matter  of  judgment  and  circumstance.  ...  It 
is  the  reader’s  responsibility  to  weigh  each  factor  according  to  his  requirements. 
No  liability  of  whatever  kind  shall  be  carried  by  the  authors.” 

The  taxonomy  is  represented  by  the  table  of  contents  of  the  guide,  reproduced  in 
part  below. 

2.  Host  and  target 

3.  Language-related  issues 

4.  User-interfacing  and  facilities 


4.1 

Compiler  invocation  and  listing  management 

4.2 

Compilation  options 

4.3 

Other  features 

4.4 

Errors  and  warnings 

4.5 

Other  software  supplied 

4.6 

Compilation  management 

5. 

Performance  and  capacity 

5.1 

Host  performance  and  capacity 

5.2 

Target  code  performance 

6. 

Compiler  and  run-time  interfacing 

6.1 

Compiler  issues 

6.2 

Run-time  system  issues 

7. 

Retargetting  and  rehosting 

7.1 

Introduction  and  definitions 

7.2 

Retargetting 

7.3 

Rehosting 

8.  Contractual  metiers 

9.  Validation 
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Chapter  2,  Host  and  Target,  briefly  treats  compiler  configuration  issues,  and  pro¬ 
vides  a  checklist  [Host  and  target  checklist  14.1] 

Chapter  3,  Language-related  issues,  extracts  from  the  Ada  language  reference 
manual  [.S'-oD  1983]  those  features  explicitly  allowed  to  vaiy  based  upon  machine 
specific  characteristics  [Machine-specific  characteristics  checklist  14.2] 

Methods: 

[Host  and  target  checklist  14.1: 

Machine-specific  characteristics  checklist  14.2] 
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4.20  WIS  COMPILER  EVALUATION  GUIDELINES 


Citations: 

[WIS  CEG  19851  “WIS  ComDiler  Evaluation  Guidelines.”  GTE  Labs.  Technical 
Report.  1985. 

Synopsis: 

This  document  presents  guidelines  that  provide  an  information  base  on  which 
specific  compiler  evaluation  methodology  and  criteria  can  be  built.  Three  types  of 
guidelines  have  been  identified:  essential  characteristics,  highly  recommended 
characteristics,  and  recommended  characteristics.  Also,  certain  questions  that 
compiler  vendors  should  be  asked  regarding  their  compilers  measurable  character¬ 
istics  are  listed.  The  guidelines  take  the  view  that  the  development  of  Ada  compil¬ 
ers  is  an  ongoing  process.  To  address  this  fact,  the  document  discusses,  where 
appropriate,  general  aspects  of  compilers,  and  specific  aspects  of  Ada  compilers. 

The  document  is  broken  down  into  four  main  sections.  Part  1  is  an  introductory 
section.  Part  2  provides  background  information  on  Ada  compilers.  Part  3  dis¬ 
cusses  compiler  architecture  issues.  Part  4  then  provides  the  main  Ada  compiler 
guidelines. 
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4.21  WIS  TOOL  EVALUATION  CRITERIA 


Citations: 

[WIS  PCEG  1985]  G.  Gicca  and  C.  Stacey,  “Proposed  Component  Evaluation 
Guiuelii  ies,  ’  GTE  Guvcmruciii  Systems,  Technical  Report,  16  August  1385. 

Synopsis: 

This  document  outlines  a  process  to  be  used  in  evaluating  currently  available  soft¬ 
ware  tools  for  inclusion  :n  an  Ada  software  development  environment.  It  defines  a 
four  phase  evaluation  process  where  each  successive  phase  takes  a  more  detailed 
view  of  the  particular  development  tool.  All  phases  have  the  same  basic  set  of 
evaluation  categories,  with  the  definition  of  each  being  defined  at  a  more  refined 
level  in  the  following  phase.  There  are  seven  such  categories.  These  are: 
“Functional  Applicability,"  “Understandability,”  “Testability,”  "Evolvability.”  “Effi¬ 
ciency,”  "Portability,”  and  “Human  Engineering.”  Their  definitions  at  a  high  ievol 
are: 


Functional  Applicability  —  the  extent  to  which  the  tool  or  component  fulfills 
a  current  need  within  a  software  development  support  environment 

Understandability  —  the  extent  to  which  the  tool  or  component  is  under¬ 
standable  from  a  systems  viewpoint 

Testability  —  the  ease  with  which  a  program  can  be  tested  to  verify  that  it 
performs  its  intended  functions 

Evolvability  —  a  category  that  evaluates  the  combination  of  both  modifiabil¬ 
ity  and  expandability 

Efficiency  —  the  amount  of  time  and  space  required  by  a  program  to  per¬ 
form  a  function 

Portability  —  the  ease  of  transferring  a  program  from  one  hardware  con¬ 
figuration  or  software  environment  to  another 

Human  Engineering  —  the  ease  of  learning,  operating,  preparing  input,  and 
interpreting  output  of  a  program. 

Each  of  the  four  phases  of  the  component  evaluation  has  its  own  rating  scheme. 

A  checklist  is  created  for  each  category  and  for  each  sub-category  within  the  ba¬ 
sic  evaluation  categories.  The  rating  scheme  itself  defines  a  set  of  numbers  from 
1  to  5.  The  reviewer  then  rates  a  particular  tool  or  component  by  assigning  a 
value  for  each  sub-category.  A  weighting  factor  is  then  used  to  prioritize  sub-cate¬ 
gories  and  main  categories.  In  the  end  a  final  set  of  numbers  is  produced  that  al¬ 
lows  for  overall  comparisons  between  tools  that  offer  similar  capabilities. 


4-24 


E&V  Guidebook,  Version  1.1 


5.  COMPILATION  SYSTEM  ASSESSORS 


For  the  purposes  of  this  document,  the  compilation  system  is  defined  as  those 
APSE  components  which  are  Ada-specific  and  are  required  for  validation:  the  compiler, 
the  code  generator,  the  program  library  management  system,  and  the  runtime  support  sys¬ 
tem.  While  each  of  these  components  have  characteristics  which  should  be  assessed  in¬ 
dividually,  the  assessment  of  their  combined  functionality  will  be  more  critical  to  the 
successful  development  of  mission  critical  software. 

The  criticality  of  assessor  development  for  these  four  compilation  system  com¬ 
ponents  is  made  evident  by  the  many  large-scale  projects  with  requirements  for  the  use  of 
Ada.  These  large-scale  projects  include  the  Strategic  Defense  Initiative  (SDI),  the  NASA 
Space  Station,  the  Software  Technology  for  Adaptable,  Reliable  Systems  (STARS)  program, 
Army  Tactical  Command  and  Control  System,  Army  WWMCCS  Information  System  (WIS), 
and  the  Advanced  Tactical  Fighter  (ATF),  Advanced  Tactical  Aircraft  (ATA),  and  Light 
Helicopter  Experimental  (LHX)  programs  being  evaluated  for  common  avion  -s  systems  un¬ 
der  the  auspices  of  the  Joint  Integrated  Avionics  Working  Group  (JIAWG).  The  successful 
performance  of  these  systems  depends  upon  the  quality/extent  of  code  generation  support 
and  execution  support  found  in  the  compilation  system. 
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5.1  Ada  COMPILER  VALIDATION  CAPABILITY  (A CVC) 

Purpose:  Validation  of  the  completeness  of  the  Ada  compiler  by  means  of  a  compiler 
test  suite.  The  ACVC  consists  of  a  test  suite,  analysis  tools,  and  accompanying 
documentation,  to  enaoie  tne  determination  of  conformance  of  Ada  compiler  imple¬ 
mentations  to  the  ANSI/MIL-STD-1815A.  Note:  The  AJPO  requires  that  Ada  compil¬ 
ers  pass  the  ACVC  and  the  vendor  allow  the  distribution  of  the  resulting  Validation 
Summary  Report  (VSR)  in  order  for  the  compiler  to  be  advertised  as  a  commercially 
available  Ada  compiler. 

[@RM:  Compilation  7. 1.6. 7,  foRM:  Completeness  6.4.9] 

Primary  References: 

l a  ACVC  1987] 

Host/OS:  Unrestricted. 

Vendors/ Agents:  [National  Technical  Information  Service] 

Method:  Automated  test  suite. 

Inputs: 

ACVC  source  code,  Ada  compiler  and  runtime  system,  and  host  (and  target)  com¬ 
puter. 

Process: 

1 .  Obtain  latest  ACVC  test  suite 

2.  Following  documentation,  compile  and  run  tests 

3.  Use  analysis  tools  on  test  outputs. 

Outputs: 

Validation  results: 

Validation  Summary  Report  (VSR). 
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5.2  IDA  BENCHMARKS 

Purpose:  Evaluation  of  the  capacity  and  performance  of  the  Ada  compiler  by  means 
of  a  compiler  test  suite. 

[@RM:  Compilation  7. 1.6. 7,  @RM:  Capacity  6.4.6; 

®RM:  Processing  Effectiveness  6.4.22;  @RM:  Storage  Effectiveness  6.4.31] 

Primary  References: 

[@IDA  1985] 

Host/OS:  VAX/VMS  or  any  system  that  can  read  ANSI  standard  tapes. 
Vendors/Agents:  [SofTech,  Inc.] 

Method:  Test  suite. 

Inputs:  IDA  source  code,  Ada  compiler  and  runtime  system,  and  host  (and  target) 
computers. 

Process: 

1 .  Obtain  test  suite  from  agent 

2.  Compile  and  run  Ada  programs. 

Outputs:  Timing  and  storage  measurements  for  individual  language  features. 
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5.3  Ada  COMPILER  EVALUATION  CAPABILITY  (ACEC) 

Purpose:  The  purpose  of  this  test  suite  is  best  stated  bv  the  following  quote  taken 
from  the  introduction  in  the  ACEC  Reader’s  Guide:  ‘‘The  ACEC  ....  consists  of  a 
portable  test  suite  and  support  tools.  ...  It  contains  test  Droblems  designed  to  meas¬ 
ure  the  execution  time  and  size  of  a  systematically  constructed  set  of  Ada  examples. 
The  support  tools  assist  the  ACEC  user  in  executing  the  test  suite  and  analyzing  the 
results  obtained.”  The  scope  of  coverage  provided  by  the  test  suite  is  shown  by  the 
following  excerpts  from  the  ACEC  classification  taxonomy: 

II.  Execution  Time  Efficiency 

A.  Language  Feature  Efficiency 

1  Required  (referenced  by  LRM  section) 

2.  Implementation  Dependent  (referenced  by  LRM  section) 

•  attributes  (LRM  Appendix  A) 

•  record  representation  clauses 

•  interrupts 

•  language  interface 

•  unchecked  programming 

B.  Pragmas 

1 .  Predefined 

2.  Implementation  Defined 

C.  Optimizations 

1.  Classical 

2.  Effects  of  Pragmas 

3.  Static  Elaboration 

•  aggregates 

•  tasks 

4.  Language  Specific 

•  Habermann-Nassi  transformation  for  tasking 

•  delay  statement  optimization 

D.  Performance  Under  Load 

1.  Task  Loading 

•  task  creation 

•  task  termination 

•  task  abortion 

•  Dining  Philosophers  Problem 

•  task  starvation 

•  task  delay 

2.  Levels  of  Nesting 

•  static 

•  dynamic 

3.  Parameter  Variation 

4.  Declarations 
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E.  Trade  Offs 

1.  Design  Issues 

•  order  of  evaluation 

•  default  vs  initialized  records 

•  order  of  selection  (rendezvous) 

•  scope  of  usage  (global,  local,  shared) 

2.  Coding  Style  Variation 

F.  Operating  System  Kernel  Efficiency 

1.  Task  Scheduling 

2.  Exception  Handling 

3.  File  I/O 

4.  Memory  Management/Storage  Reclamation 

5.  Elaboration 

6.  Run  Time  Checks 

7.  Interrupt  Handling 

G.  Application  Profile  Tests 

1 .  Classical 

•  Whetstone 

•  Dhrystone 

•  Ackerman’s 

•  Computer  Family  Architecture 

•  Sort  Variations 

•  Math  Applications 

•  Livermore  Loops 

•  Knuth  Loops 

2.  Ada  in  Practice 

•  E-3A  simulator 

•  navigation  algorithms 

•  radar  tracking  algorithms 

•  communication  algorithms 

•  electronic  warfare 

•  avionics 

3.  Ideal  Ada 

•  Al  applications 

•  data  encryption 

III.  Code  Size  Efficiency 

A.  Expansion  Code  Size 

B.  Run  Time  System  Size 

C.  Executable  File  Size 

The  first  version  of  the  ACEC  is  expected  in  August  1988. 

[®RM:  Compilation  7. 1.6. 7,  @RM:  Processing  Effectiveness  6.4.22; 

'aRM:  Storage  Effectiveness  6.4.31] 

Primary  References; 

[ACEC  1988]  “Ada  Compiler  Evaluation  Capability  (ACEC)  Technical  Operating 
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Report  (TOR)  Reader's  Guide,"  Air  Force  Wright  Aeronautical  Laboratory,  Docu¬ 
ment  Number  D500-1 1790-2,  10  August  1988. 

Host/OS:  Portable 

Vendors/Agents:  [DACS] 

Method: 

Automated  test  suite 

Inputs:  ACEC  source  code,  Ada  compiler  and  runtime  system,  host  and  target 
computer. 

Process: 

1 .  Obtain  the  ACEC 

2.  Compile  and  run  the  tests  according  to  the  documentation 

3.  Use  the  analysis  tools  on  the  test  outputs. 

Outputs:  Reports  containing  the  execution  time  and/or  code  size  for  the  selected 
tests. 
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5.4  PIWG  BENCHMARK  TESTS 

Purpose:  Identification  of  performance  characteristics  of  Ada  compilers.  The  tests 
examine  the  performance  of  the  compiler  itself  in  terms  of  compilation  speed,  as  well 
as  the  quality  of  the  generated  code  for  both  processing  and  storage  effectiveness. 
The  test  suite  measures  performance  for  both  isolated  language  features  and  com¬ 
posites  or  mixes  of  language  features  (using  the  Whetstone  and  Dhrystone  tests). 
[@RM:  Compilation  7. 1.6. 7.  @RM:  Processing  Effectiveness  6.4.22; 

@RM:  Storage  Effectiveness  6.4.31] 

Primary  References: 

Host/OS:  Unrestricted 

Vendors/Agents:  [PIWG] 

Method: 

Automated  test  suite. 

Inputs:  PIWG  source  code,  Ada  compiler  and  runtime  system,  and  host  (and  tar¬ 
get)  computer. 

Process: 

1.  Obtain  the  latest  PIWG  tests 

2.  Compile  and  run  tests  according  to  the  documentation. 

Outputs:  Reports  on  the  outcome  of  each  test  run. 
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5.5  UNIVERSITY  OF  MICHIGAN  BENCHMARK  TESTS 

Purpose:  Identification  of  the  execution  efficiency  of  the  code  generated  by  Ada 
compilers.  The  tests  measure  only  the  performance  of  isolated  language  features  as 
they  relate  to  real-time  performance. 

[@RM:  Compilation  7. 1.6. 7,  @RM:  Processing  Effectiveness  6.4.22] 

Primary  References: 

[Mich  1986]  R.M.  Clapp,  L.  Duchesneau,  R.A.  Volz.  T.N.  Mudge,  and 

T.  Schultze,  “Toward  Real-Time  Performance  Benchmarks  for  Ada.”  Electrical  En¬ 
gineering  and  Computer  Science  Dept.,  Univ.  of  Michigan,  RSD-TR-6-86,  January 
1986,  pp.  1-25. 

Host/OS:  Unrestricted 

Vendors/Agents:  [UMich] 

Method: 

Test  suite. 

Inputs:  UMichigan  source  code,  Ada  compiler  and  runtime  system,  and  host 
(and  target)  computer. 

Process: 

1.  Obtain  the  UMichigan  tests 

2.  Compile  and  run  according  to  the  documentation. 

Outputs:  Reports  on  the  outcome  of  each  test  run 
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5.6  MITRE  BENCHMARK  GENERATOR  TOOL  (BGT) 

Purpose:  Evaluation  of  the  ability  of  an  Ada  compilation  system  to  support 
development  of  very  large  systems  in  Ada.  Under  sponsorship  of  the  Federal  Avia¬ 
tion  Administration,  MITRE  developed  the  Benchmark  Generator  Tool  (BGT).  The 
benchmark  tests  address  capacity  issues  arising  with  large  system  developments. 

The  initial  version  (1988)  includes  two  types  of  tests:  Library  Capacity  Tests  and  De¬ 
pendency  Maintenance  Tests. 

[@RM:  Compilation  7. 1.6.7,  @RM:  Capacity  6.4.6] 

Primary  References: 

[MITRE  BGT  1986]  S.R.  Rainer,  and  T.P.  Reagan,  “User’s  Manual  for  the  Ada 
Compilation  Benchmark  Generator  Tool,”  MITRE  Corporation,  MTR-87W001 92-01, 
January  1988. 

Host/OS:  Any  for  which  an  Ada  compiler  exists. 

Vendors/ Agents:  [M!TRE,  McLean,  VA] 

Method: 

Automated  tool. 

Inputs: 

BGT  source  code,  Ada  compiler,  and  host  computer. 

Process: 

1.  Obtain  the  BGT 

2.  Compile  according  to  the  documentation. 

Outputs: 

Results  of  the  above  analysis,  including  capacity  limits,  link  time, 
compilation  time,  etc. 
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5.7  UK  Ada  EVALUATION  SYSTEM  (AES) 

Purpose:  Evaluation  of  Ada  compilers  and  associated  linkers/loaders,  program 
library  systems,  debuggers,  and  run-time  libraries.  A  test  suite  and  a  methodology 
(AES)  were  developed  by  Software  Sciences  Ltd.,  under  sponsorship  of  the  UK  Minis¬ 
try  of  Defense  (MoD).  The  British  Standards  Institute  (BSI)  has  been  sponsored  by 
the  MoD  to  provide  an  Ada  Evaluation  Service,  using  the  AES.  Interested  parties, 
such  as  compiler  vendors  or  potential  compiler  purchasers,  may  pay  BSI  to  conduct 
an  evaluation  or  to  supply  a  copy  of  an  existing  evaluation  report. 

Primary  References: 

[UK  AES  1986] 

Host  OS:  Any  for  which  an  Ada  compiler  exists. 

Vendors/ Agents:  [BSI,  Milton-Keynes,  UK] 

Method: 

Automated  test  suite  and  questionnaire. 

Inputs: 

AES  source  code  and  questionnaire,  Ada  compiler  and  runtime  system,  and 
host  (and  target)  computer. 

Process: 

Pay  BSI  to  do  an  evaluation  or  purchase  an  existing  evaluation  report. 

Outputs: 

An  Evaluation  Report  organized  in  a  standard  format. 
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5.8  COMPILATION  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  compilation  by  developing  a  list  of  functional 
capabilities. 

[@RM:  Compilation  7. 1.6. 7,  @RM:  Power  6.4.21] 

Primary  References: 

[@E&V  Schema  1987:  B.; 

Classification  Schema/E&V  Taxonomy  Checklists:  4.4] 

Vendors/ Agents:  [E&V  Team] 

Method:  Capabilities  checklist 

Inputs:  Capability  checklist  (see  Table  5.8-1)  and  compiler  documentation. 

Process:  Check  off  capabilities  demonstrated  during  compiler  runs  or  dis¬ 
cussed  in  the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  compiler. 


5-11 


E&V  Guidebook,  Version  1.1 


TABLE  5.8-1 

COMPILATION  CAPABILITIES  CHECKLIST 

FEATURE  FOUND 

Conditional  Compilation 

Debug  Information  Generation 

Enable/Disable  Listing 

Errors  Only  Listing 

Error  Identification 

Set  Default  Directory  For  Source 

Set  Listing  Width  And  Height 

Specify  Different  Program  Library 

Specify  Main  Program 

Disable  Use  Of  SYSTEM  Library 

Suppress  All  Run-Time  Checks 

Compile  Multiple  Files 

Language  Sensitive  Editor  Support 

Specify  Error  Limit 

Enable/Disable  An  Error  Category 

Specify  Optimization  Parameters 

Syntax  Only  Checking 

Symbol  Table 

Variable  Set/Use  Indications  (Cross  Reference) 

Object  Code  Listing 
Object  Attribute  Map 
Code  Statistics 

Unidentified  Compiler  Options  (Pragmas) 

Controlled  Dynamic  Storage 

Elaboration  Control 

Inline  Expansion  Of  Subprograms 

Interface  With  Other  Languages 

Specify  Memory  Size 

Pack  Data  Representations  In  Memory 

Priority  Control  Of  Concurrent  Tasks 

Shared  Variables 

Specify  Storage  Unit 

Specify  Alternative  System  Characteristics 
Machine  Code  Mapping 
Machine  Code  Insertions 
Cross  Compilation 
Error  Reporting 
Exceptions  List 
Identify  Target  Dependencies 
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5.9  PROGRAM  LIBRARY  MANAGEMENT  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  program  library  management  by  developing  a 
list  of  functional  capabilities. 

[<a>RM:  Program  Library  Management  7.2. 1.7,  @RM:  Power  6.4.21] 

Primary  References: 

[@E&V  Schema  1987:  B. : 

Classification  Schema/E&V  Taxonomy  Checklists:  4.4] 

Vendors/Agents:  [E&V  Team] 

Method:  Capabilities  checklist 

Inputs:  Capabilities  checklist  (see  Table  5.9-1)  and  program  library  manager 
documentation. 

Process:  Check  off  capabilities  demonstrated  by  the  program  library  manager 
or  discussed  in  the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  program  library  manager. 

TABLE  5.9-1 

PROGRAM  LIBRARY  MANAGEMENT  CAPABILITIES  CHECKLIST 

FEATURE  FOUND 

Listing  Information 

List  Of  Logical  Unit  Names 
Associated  File  Names  For  Unit 
Units  Using  Specified  Unit 
Units  Used  By  Specified  Unit 
Size  Information 
Time-Stamp  Information 
Kind  Of  Compilation  Unit 
Units  Used  To  Construct  Executable 
Completeness  And  Currency  Check 
Automatic  Recompilation 
Spawn  CLI  Subprocess 
Create  Structures 
Move  Elements  Between  Libraries 
Move  Elements  Between  Directories 
Remove  Compilation  Unit 
Library  Access  Control 
Read  Only  (Shared) 

Exclusive 
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5.10  ARTEWG  CATALOGUE  OF  Ada  RUNTIME  IMPLEMENTATION  DEPENDENCIES 

Purpose:  The  purpose  of  this  document  is  best  stated  by  the  following  quotation  taken 
from  the  rationale  section  in  the  catalogue:  "The  main  goal  of  this  catalogue  is  to  be 
the  one  place  where  all  the  areas  of  the  Ada  Reference  Manual  ...  which  permit  im¬ 
plementation  flexibilities  can  be  found.”  These  implementation  dependencies  affect 
the  performance  and  adaptation  characteristics  of  the  generated  code.  The  text  de¬ 
scribes  each  known  dependency  by  a  number  (which  identifies  the  relevant  section 
and  paragraph  in  the  Ada  Reference  Manual),  a  topic  or  title,  a  question  which  poses 
the  implementation  issue,  a  dependency  type  (either  explicit  or  implicit),  a  rationale 
explaining  why  the  dependency  exists,  and  an  Ada  example  to  further  clarify  the  de¬ 
pendency.  (These  descriptions  could  be  used  as  the  basis  for  an  automated  test 
suite.) 

[®RM:  Compilation  7. 1.6. 7,  @RM:  Anomaly  Management  6.4.2; 
i&RM:  Processing  Effectiveness  6.4.22;  @RM:  Retargetability  6.4.27; 
rSRM:  Storage  Effectiveness  6.4.31] 

Primary  References: 

[ARTEWG  1987]  Catalogue  of  Ada  Runtime  Implementation  Dependencies,” 
Association  for  Computing  Machinery,  Special  Interest  Group  on  Ada, 

Ada  Runtime  Environment  Working  Group,  1  December  1987. 

Host/OS:  Not  applicable. 

Vendors/Agents:  [ARTEWG] 

Method: 

Questionnaire. 

Inputs:  Descriptions  of  implementation  dependent  features. 

Process: 

1 .  Select  critical  dependencies 

2.  Build  and  run  tests  for  each  dependency  or  ask  vendor  how 
dependencies  are  implemented 

3.  Select  compiler  and/or  make  coding  standards  based  on  results  of  step  2. 

Outputs:  Evidence  showing  how  features  are  implemented. 
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5.11  ARTEWG  RUNTIME  ENVIRONMENT  TAXONOMY 

Purpose:  Describes  the  basic  elements  of  Ada  runtime  environments  and  provides  a 
common  vocabulary.  The  following  excerpt  is  taken  from  the  introduction  to  the  Tax¬ 
onomy  section.  “If  a  runtime  environment  for  an  Ada  program  is  composed  of  a  set 
of  data  structures,  a  set  of  conventions  for  the  executable  code,  and  a  collection  of 
predefined  routines,  then  the  question  arises:  what  are  examples  of  these  elements, 
and  moreover,  what  is  the  complete  set  from  which  such  elements  are  taken  when  a 
particular  runtime  environment  is  built?. ..It  should  be  noted  that  the  dividing  line  be¬ 
tween  the  predefined  runtime  support  library  on  one  hand,  and  the  conventions  and 
data  structures  of  a  compiler  on  the  other  hand,  is  not  always  obvious.  One  Ada  im¬ 
plementation  may  use  a  predefined  routine  to  implement  a  particular  language  fea¬ 
ture,  while  another  implementation  may  realize  the  same  feature  through  conventions 
for  the  executable  code.  ...  This  taxonomy  concerns  itself  primarily  with  those  as¬ 
pects  of  the  runtime  execution  architecture  which  are  embodied  as  routines  in  the 
runtime  library.  It  does  not  treat  issues  of  code  and  data  conventions,  nor  issues  re¬ 
lated  to  particular  hardware  functionalities,  in  any  great  depth.” 

[@RM:  Runtime  Environment  7. 2. 3. 5,  @RM:  Power  6.4.21] 

Primary  References: 

[ARTEWG  1988]  "A  Framework  for  Describing  Ada  Runtime  Environments,”  Pro¬ 
posed  by  Ada  Runtime  Environment  Working  Group  (SIGAda),  Ada  Letters,  Volume 
VIII,  Number  3,  May/June  1988,  pp.  51-68. 

Vendors/Agents:  [ARTEWG] 

Method:  Capabilities  checklist 

Inputs:  Capability  checklist  (see  Table  5.11-1)and  runtime  environment 
documentation. 

Process:  Check  off  capabilities  demonstrated  by  the  runtime  environment  or  dis¬ 
cussed  in  the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  runtime  environment. 
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TABLE  5.11-1 

RUNTIME  ENVIRONMENT  TAXONOMY 


FEATURE 

FOUND 

Runtime  Execution  Model 

Dynamic  Memory  Management 

Processor  Management 

Interrupt  Management 

Time  Management 

Exception  Management 

Rendezvous  Management 

Task  Activation 

Task  Termination 

I/O  Management 

Commonly  Called  Code  Sequences 

Target  Housekeeping  Functions 
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6.  TARGET  CODE  GENERATION  AIDS  AND  ANALYSIS 

TOOLSET  ASSESSORS 


These  tools  are  used  to  assess  host-target  system  cross-assemblers;  host- 
based  target  linkers  and  loaders;  host-based  target  system  instruction-level  simulators/emu¬ 
lators;  host-based  target-code  symbolic  debuggers;  and  host-based  target  system 
instrumentation  interfaces  which  provide  visibility  into  target  processes  during  program  exe¬ 
cution.  These  assessments  are  also  used  in  the  case  where  the  host  computer  is  also  the 
target  computer. 
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6.1  ASSEMBLING  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  assembling  by  developing  a  list  of  functional 
capabilities. 

[rgpRM:  Assembling  7. 1.6. 6,  foRM:  Power  6.4.21] 

Primary  References: 

Vendors/Agents:  [E&V  Team] 

Method;  Capabilities  checklist 

Inputs-  Capabilities  checklist  (see  Table  6.1-1)  and  assembler  documentation. 

Process:  Check  off  capabilities  demonstrated  during  assembler  runs  or  discussed 
in  the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  assembler. 


TABLE  6.1-1 

ASSEMBLING  CAPABILITIES  CHECKLIST 


FEATURE 

FOUND 

Code  Generation 

Macro  Preprocessing 

Conditional  Assembly 

Debug  Information  Generation 

Enable/Disable  Listing 

Errors  Only  Listing 

Set  Listing  Width  and  Height 

Suppress  All  Run-Time  Checks 

Assemble  Multiple  Files 

Specify  Error  Limit 

Enable/Disable  An  Error  Category 

Syntax  Only  Checking 

Symbol  Table 

Code  Statistics 

Cross  Assembly 
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6.2  LINKING/LOADING  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  linking/loading  by  developing  a  list  of  functional 
capabilities. 

[@RM:  Unking/Loading  7.1.6.13,  @RM:  Power  6.4.21] 

Primary  References: 

[@E&V  Schema  1987:  B.; 

Classification  Schema/E&V  Taxonomy  Checklists:  4.4] 

Vendors/ Agents:  [E&V  Team] 

Method:  Capabilities  checklist 

Inputs:  Capabilities  checklist  (see  Table  6.2-1)  and  linker/loader  documentation. 

Process:  Check  off  capabilities  demonstrated  during  linker/loader  runs  or  dis¬ 
cussed  in  the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  linker/loader. 

TABLE  6.2-1 


LINKING/LOADING  CAPABILITIES  CHECKL 

ST 

FEfi TURE 

FOUND 

Non-Specific  Language  Linking 

Deferred  (After  A  Specific  Time) 

Enable/Disable  Link  Map  Generation 

Specify  Full/Brief  Link  Map 

Generate  A  Link  Command  File 

Enable/Disable  Symbol  Cross-Reference 

Generate  Debug  Information 

Enable/Disable  Execution  File  Creation 

Specify  Batch/Nobatch  Operation 

Specify  Map  File  Name 

Specify  Object  File  Name 

Specify  Diagnostic  Output  File 

Enable/Disable  System  Library  Search 

Enable/Disable  Traceback  Information 

Library  Search  Capabilities 

Extended  Options  Capabilities 

Sharable  Image  Support 

Specify  Maximum  Memory 

Specify  Optimization  Parameters 

Force  Load 

Enable/Disable  Library  Trace 

Specify  Main  Program 

Non-Specific  Language  Main  Program 

Overlays 
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6.3  IMPOrT /EXPORT  CAPABILITIES  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  import/export  by  developing  a  list  of  functional 
capabilities. 

['a.RM:  Import  Export  7. 2. 3. 6,  rSRM:  Power  6.4.21] 

Primary  References: 

[foE&V  Schema  1987:  B.; 

Classification  Schema  E&V  Taxonomy  Checklists:  4.4] 

Vendors.  Agents:  [E&V  Team] 

Method:  Capabilities  checklist 

inputs:  CaDabilities  checklist  (see  Table  6.3-1)  and  import/export  documentation. 

Process:  Check  off  capabilities  demonstrated  by  the  irriDort/export  system  or  dis¬ 
cussed  in  the  documentation. 

Outputs:  A  list  of  capacities  performed  by  the  import/export  system. 


TABLE  6.3-1 

IMPORT  ^EXPORT  CAPABILITIES  CHECKLIST 


FEATURE 

FOUND 

Host  to  Target  Object  Downloading 

Target  to  Host  Data  Uploading 

Note.  This  table  will  be  expanded  in  a  future  version  of  the  Guidebook. 
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6.4  EMULATION  CAPABILITIES  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  emulation  by  developing  a  list  of  functional 
capabilities. 

[@RM:  Emulation  7.3.2.13,  @RM:  Power  6.4.21] 

Primary  References: 

[@E&V  Schema  1987:  B.; 

Classification  Schema/E&V  Taxonomy  Checklists:  4.4] 

Vendors/Agents:  [E&V  Team] 

Method:  Capabilities  checklist 

Inputs:  Capabilities  checklist  (see  Table  6.4-1)  and  emulation  system  documenta¬ 
tion. 

Process:  Check  off  capabilities  demonstrated  by  the  emulation  system  or  dis¬ 
cussed  in  the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  emulation  system. 


TABLE  6.4-1 

EMULATION  CAPABILITIES  CHECKLIST 


FEATURE 

FOUND 

Hardware 

System  Emulation/Simulation 

Note:  This  table  will  be  expanded  in  a  future  version  of  the  Guidebook. 
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6.5  DEBUGGING  CAPABILITIES  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  debugging  by  developing  a  list  of  functional 
capabilities. 

[raRM:  Debugging  7. 3. 2. 5,  foRM:  Power  6.4.21] 

Primary  References: 

[®E &V  Schema  1987:  B.; 

Classification  Schema/E&V  Taxonomy  Checklists:  4.4] 

Vendors/Agents:  [E&V  Team] 

Method:  Caoabilities  checklist 

inputs:  Capabilities  checklist  (see  Table  6.5-1)  and  debugger  documentation. 

Process:  Check  off  capabilities  demonstrated  by  the  debugger  or  discussed  in 
the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  debugger. 
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TABLE  6.5-1 

DEBUGGING  CAPABILITIES  CHECKLIST 


FEATURE 

FOUND 

Instrumentation 

Statement 

Branch 

Unit 

CSC 

Machine  Level  Debugging 

Symbolic  Debugging 

Tracing 

Breakpoint  Control 

Data  Flow  Tracing 

Path  Flow  Tracing 

Selectable  Level  Of  Granularity 

Display 

Program  Source 

History 

Stack 

Tasks 

Breakpoints 

Tracepomts 

Memory 

Collections  And  Global  Heaps 

Name  Of  Current  Exception 

Evaluate  Objects 

Step 

Single 

By  Discrete  Amounts 

Into  Subprograms 

Over  Subprograms 

To  Next  Scheduling  Event 

To  Next  Exception 

To  End  of  Program  Unit 

Miscellaneous 

Symbol  Abbreviation 

Set  Context  For  Control 

Input  Debugger  Command  Files 

Modify  Variable  Values 

Modify  Object  Code 

Modify  Control  Flow 

Console  interrupt 

Full  Screen  Mode 

Keypad  For  Entering  Commands 

Virtual  Clock 

Special  Compilation  Mode 

Multi-Language  Support 

Dynamic  Interrupt 

Optimization  Support 

Units  Comprising  Executable 
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6.6  TIMING  ANALYSIS  CAPABILITIES  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  timing  analysis  by  developing  a  list  of  functional 
capabilities. 

[®RM:  Timing  7.3.2.14.  @RM:  Power  6.4.21] 

Primary  References: 

Vendors/ Agents:  [E&V  Team] 

Method:  Capabilities  checklist 

Inputs:  Capabilities  checklist  (see  Table  6.6-1)  and  timing  analysis  system  docu¬ 
mentation. 

Process:  Check  off  capabilities  demonstrated  by  the  timing  analysis  system  or 
discussed  in  the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  timing  analysis  system. 


TABLE  6.6-1 

TIMING  ANALYSIS  CAPABILITIES  CHECKLIST 


FEATURE 

FOUND 

Timing  Instrumentation 

Obtrusive 

Non-Obtrusive 

Fraction  By  Section  Of  Code 

Tasking  Monitor 

Fraction  Executing 

Fraction  Runnable 

Fraction  Runnable  And  Not  Executing 

Time  Between  Runnable  And  Executing 

Time  Between  Events 

System  Idle  Time 
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6.7  TUNING  ANALYSIS  CAPABILITIES  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  tuning  analysis  by  developing  a  list  of  functional 
capabilities. 

[@RM:  Tuning  7.3.2.15.  @RM:  Power  6.4.21] 

Primary  References: 

[@E&V  Schema  1987:  B.; 

Classification  Schema/E&V  Taxonomy  Checklists:  4.4] 

Vendors/ Agents:  [E&V  Team] 

Method:  Capabilities  checklist 

Inputs:  Capabilities  checklist  (see  Table  6.7-1)  and  tuning  analysis  system  docu 
mentation. 

Process:  Check  off  capabilities  demonstrated  by  the  tuning  analysis  system  or 
discussed  in  the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  tuning  analysis  system. 


TABLE  6.7-1 

TUNING  ANALYSIS  CAPABILITIES  CHECKLIST 


FEATURE 

FOUND 

Task  Monitoring  Instrumentation 

Note:  This  table  will  be  expanded  in  a  future  version  of  the  Guidebook. 
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6.8  REAL-TIME  ANALYSIS  CAPABILITIES  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  real-time  analysis  by  developing  a  list  of  functional 
capabilities. 

[fg-RM:  Real-Time  Analysis  7.3.2.17,  @RM:  Power  6.4.21] 

Primary  References: 

[@E&V  Schema  1987:  B.; 

Classification  Schema/E&V  Taxonomy  Checklists:  4.4] 

Vendors-  Agents:  [E&V  Team] 

Method:  Capabilities  checklist 

inputs:  Capabilities  checklist  (see  Table  6.8-1)  and  real-time  analysis  system 
documentation. 

Process.  Check  off  capabilities  demonstrated  by  the  real-time  analysis  system  or 
discussed  in  the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  real-time  analysis  system. 


TABLE  6.8-1 

REAL-TIME  ANALYSIS  CAPABILITIES  CHECKLIST 


FEATURE 

FOUND 

Hardware-ln-The-Loop 

Non-lntrusive  Instrumentation 

Performance  Analysis 

Symbolic  Trace 
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7.  TEST  SYSTEMS  ASSESSORS 

These  assessors  examine  the  ability  of  the  APSE  or  APSE  component  to  sup¬ 
port  and  facilitate  the  planning,  development,  execution,  evaluation,  and  documentation  of 
tests  of  software. 


7.1  TESTING  CAPABILITIES  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  testing  by  developing  a  list  of  functional  capabilities. 
[@RM:  Testing  7.3.2.10,  @RM:  Power  6.4.21] 

Primary  References: 

[@DeMillo  1986]  R.A.  DeMillo,  “Functional  Capabilities  of  a  Test  and  Evaluation 
Subenvironment  in  an  Advanced  Software  Engineering  Environment,”  Georgia 
Institute  of  Technology  GIT-SERC-86/07,  20  October  1986. 

Vendors/Agents:  [GIT] 

Method:  Capabilities  checklist 

Inputs:  Capabilities  checklist  (see  Table  7.1-1)  and  testing  system  documentation. 

Process:  Check  off  capabilities  demonstrated  by  the  testing  system  or  discussed  in 
the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  testing  system. 
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TABLE  7.1-1 

TESTING  CAPABILITIES  CHECKLIST 


FEATURE 

FOUND 

Static  Analyzers 

Code  Auditors 

Traceability  Analyzers 

Consistency  Checkers 

Interface  Analyzers 

Completeness  Checkers 

Tool  Building  Services 

Common  “Front-End"  Facilities  for  Lanmmges  of  Interest  (Parsing, 
Source  &  Internal  Form  Manipulate  cution  Facilities) 

Tool  Composition  Aids 

Test  Building  Services  (including  Test  Data  Generators) 

Symbolic  Evaiuators 

Component  Coverage  Analyzers 

Data  Flow  Analyzers 

Assertion  Processors 

Mutation  Analyzers 

Path  and  Domain  Selection  Aids 

Random  Test  Generators 

Test  Description  and  Preparation  Services 

Data  Editors 

Data  Auditors 

Body/Stub  Generators 

File  Comparators 

Data/File  Services 

Software  and  System  Test  Communications  Facilities 

Test  Execution  Services 

Test  Harness  Generator 

Data  and  Error  Logging  Services 

Quality  Measurement  Tools 

Test  Analysis  Services 

Correctness  Analyzers  (Oracles) 

Instrumentation  Aids 

Status  Display  Tools 

Data  Reduction  and  Analysis  Tools 

Decision  Support  Services 

Documentation  Services 
information  Repositories 

Problem  Report  Processing  and  Analysis  Tools 

Change  Request  Processing  and  Analysis  Tools 
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7.2  SEI  UNIT  TESTING  AND  DEBUGGING  EXPERIMENT 

Purpose:  Evaluation  of  an  environment’s  capabilities,  from  the  point  of  view  of  the 

unit  tester.  An  experiment  was  designed  to  simulate  the  activities  normally  associated 
with  smaii  projects,  namely  the  design,  creation,  modification,  and  testing  of  a  single 
unit  or  module.  See  also  the  SEI  Design  Support  Experiment  [9.1]. 

[@RM:  Debugging  7. 3. 2. 5,  @RM:  Power  6.4.21; 

@RM:  Debugging  7. 3. 2. 5,  @RM:  Processing  Effectiveness  6.4.22; 

@RM:  Testing  7.3.2.10,  @RM:  Power  6.4.21; 

@RM:  Testing  7.3.2.10,  @RM:  Processing  Effectiveness  6.4.22] 

Primary  References: 

[raweiderman  1987,  Chapter  6; 

Weiderman:  Evaluation  of  Ada  Environments,  4.13] 

Host/OS:  VAX/VMS  and  VAX/UNIX 

Vendors/Agents:  [SEI] 

Method:  Structured  experiment 

Inputs:  The  “generic”  experiment  description,  an  APSE,  and  host  (and  target)  com¬ 
puter. 

Process:  "Instantiate”  the  experiment  for  a  specific  Host/OS/APSE  combination  and 
carry  it  out. 

Outputs:  A  filled-in  table  of  functional  elements  present  and  missing,  elapsed  time  and 
cpu  time  values,  and  subjective  judgments  based  on  the  experience. 
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8.  TOOL/HOST  INTERFACE  ASSESSORS 


These  assessors  evaluate  or  validate  implementations  of  Ada  language  specifi¬ 
cations  for  tool/host  interface  sets.  Interface  sets  that  may  be  assessed  could  include  a 
CAIS  or  a  CAIS-A  implementation.  Portable  Common  Tool  Environment  (PCTE)  implementa¬ 
tions  and  Ada  language  interfaces  to  the  UNIX  operating  system  and  its  variants  (e.g., 
Berkeley  UNIX.  System  V,  A/UX,  POSIX). 


8.1  CAIS  IMPLEMENTATION  VALIDATION  CAPABILITY  (CIVC) 

Purpose:  Assess  the  conformance  of  CAIS  implementations  to  the  DoD-STD-1838 
and  DoD-STD-1838 A  standards.  The  CIVC  consists  of  a  test  suite,  analysis  tools, 
and  associated  documentation  which  enable  validators  and  CAIS  implementors  to 
determine  the  completeness  of  CAIS  implementations  with  respect  to  conformance 
to  the  standards.  A  suite  of  tests  to  be  compiled  and  executed  with  interfaces 
provided  for  a  CAIS  implementation.  Analysis  tools  are  utilized  for  aiding  the  users 
in  selecting  tests  and  obtaining  results. 

[@RM:  Kernel  7. 2. 3. 3,  @RM:  Completeness  6.4.9] 

Primary  References:  [CIVC  1988]  “CIVC  I  Implementor’s  Guide,”  Air  Force  Wright  Aeronau¬ 
tical  Laboratory.  CVC-VREL-19,  Draft  August  1988. 

Host/OS: 

Not  Applicable 

Vendors/Agents: 

TBA 

Method:  Automated  test  suite 

Inputs:  The  CIVC  test  suite,  CAIS  implementation,  Ada  compiler  and  runtime  system, 
and  host  computer, 

Process: 

1 .  Obtain  the  CIVC  test  suite 

2.  Compile  and  run  the  tests 

3.  Collect  and  analyze  the  results. 

Outputs:  Report  describing  the  conformance  of  various  aspects  of  the  CAIS  imple¬ 
mentation  to  DoD-STD-1838  or  1838A. 
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8.2  TOOL  SUPPORT  INTERFACE  EVALUATION 

Purpose:  Evaluation  of  tool  support  interfaces  in  terms  of  four  criteria:  level, 

appropriateness,  implementability,  and  performance.  Five  “scenarios”  were  designed 
and  used  to  exercise  a  prototype  CAIS  implementation  and  a  prototype  PCTE  imple¬ 
mentation.  The  scenarios  involved  a  configuration  management  system,  an  edit-com¬ 
pile-link-test  cycle,  a  conference  management  system,  a  window  manager,  and  a 
design  editor. 

[@RM:  Kernel  7.2. 3. 3] 

Primary  References: 

[Long  1988]  F.W.  Long,  and  M.D.  Tedd,  "Evaluating  Tool  Support  Interfaces," 

Ada  in  Industry.  Proceedings  of  the  Ada-Europe  Conference,  Munich,  7-9  June  1988, 
Cambridge  University  Press,  1988. 

Host/OS:  Sun 

Vendors/Agents:  [College  of  Wales,  UK] 

Method:  Structured  experiment 

Inputs:  The  source  code  for  the  scenarios,  the  tool  support  interface(s)  (CAIS,  PCTE, 
other),  Ada  compiler  and  runtime  system,  and  host  computer. 

Process: 

1 .  Obtain  the  source  code  for  the  scenarios 

2.  Compile  and  run  the  scenario(s) 

3.  Collect  the  results. 

Outputs:  Objective  results  and  subjective  conclusions  concerning  the  impact  on  tool 
writers  and  the  cost  and  behavior  of  the  interface  implementation. 
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9.  REQUIREMENTS/DESIGN  SUPPORT  ASSESSORS 


These  assessors  measure  the  suitability  and  effectiveness  of  various  software 
definition,  specification,  and  design  tools.  This  specifically  includes  assessors  of  Ada  Pro¬ 
gram  Design  Language  (PDL)  implementations  and/or  guidelines  in  the  use  of  Ada  as  a 
PDL. 


9.1  SEI  DESIGN  SUPPORT  EXPERIMENT 

Purpose:  Evaluation  of  the  design  a:.d  code  development  capabilities  of  an  environment, 
as  represented  in  a  small  project.  An  experiment  was  designed  to  simulate  the  activi¬ 
ties  normally  associated  with  small  projects,  namely  the  design,  creation,  modification, 
and  testing  of  a  single  unit  or  module.  See  also  the  SEI  Unit  Testing  and  Debugging 
Experiment  [7.2]. 

[@RM:  Preliminary  Design  7. 1.6. 4,  (a/RM:  Power  6.4.21; 

@RM:  Preliminary  Design  7. 1.6. 4,  @RM:  Processing  Effectiveness  6.4.22; 

@RM:  Preliminary  Design  7. 1.6. 4,  @RM:  Storage  Effectiveness  6.4.31; 

@RM:  Detailed  Design  7. 1.6. 5,  @RM:  Power  6.4.21; 

@RM:  Detailed  Design  7. 1.6. 5,  @RM:  Processing  Effectiveness  6.4.22; 

@RM: .  Detailed  Design  7. 1.6. 5,  @RM:  Storage  Effectiveness  6.4.31] 

Primary  References:  [@Weiderman  1987,  Chapter  5; 

Weiderman:  Evaluation  of  Ada  Environments,  4.13] 

Host/OS:  VAX/VMS  and  VAX/UNIX 

Vendors/Agents:  [SEI] 

Method:  Structured  experiment. 

Inputs:  The  "generic”  experiment  description,  an  APSE,  and  host  computer. 

Process:  "Instantiate”  the  experiment  for  a  specific  Host/OS/APSE  combination  and 
carry  it  out. 

Outputs:  A  filled-in  checklist  of  functional  elements  present  and  missing,  tables  of 
time  and  space  data,  and  subjective  judgments  based  on  the  experience. 
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10.  CONFIGURATION  MANAGEMENT  SUPPORT  ASSESSORS 


These  assessors  examine  the  performance,  usability,  and  completeness  of  the 
APSE  or  APSE  component  functionality  related  to  controlling  the  contents  of  software  sys¬ 
tems.  This  includes  monitoring  the  status,  preserving  the  integrity  of  released  and  develop¬ 
ing  versions,  and  controlling  the  effects  of  changes  throughout  the  lifetime  of  the  software 
system. 


10.!  CONFIGURATION  MANAGEMENT  CAPABILITIES  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  configuration  management  by  developing  a  list  of 
functional  capabilities. 

[raRM:  Configuration  Management  7. 2. 2. 7,  r®RM:  Power  6.4.21] 

Primary  References. 

[rSE&V  Schema  1987:  B.; 

Classification  Schema.  E&V  Taxonomy  Checklists:  4.4] 

Vendors/Agents:  [E&V  Team] 

Method:  Capabilities  checklist 

Inputs:  Capabilities  checklist  (see  Table  10.1-1)  and  configuration  management  docu¬ 
mentation. 

Process:  Check  off  capabilities  demonstrated  by  the  configuration  manager  or  dis¬ 
cussed  m  the  documentahon. 

Outputs:  A  list  of  capabilities  performed  ty  the  configuration  manager. 
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10.2  SEI  CONFIGURATION  MANAGEMENT  EXPERIMENT 

Purpose:  Evaluation  of  the  configuration  management  and  version  control  capabilities  of  an 
environment.  An  experiment  was  designed  to  simulate  the  system  integration  and  test¬ 
ing  phase  of  the  life  cycle  by  having  three  separate  development  lines  of  descent  from 
a  single  baseline. 

[raRM:  Configuration  Management  1. 2.2.1,  ®RM:  Power  6.4.21; 

@RM:  Processing  Effectiveness  6.4.22] 

Primary  References:  [I'SWeiderman  1987,  Chapter  3; 

Weiderman:  Evaluation  of  Ada  Environments,  4.13] 

Host/OS:  VAX/VMS  and  VAX/UNIX 

Vendors/Agents:  [SEI] 

Method:  Structured  experiment. 

Inputs:  The  "generic''  exoeriment  description,  an  APSE,  and  host  computer. 

Process:  “Instantiate"  the  experiment  for  a  specific  Host/OS/APSE  combination  and 
carry  it  out. 

Outputs:  A  filled-in  checklist  showing  functional  elements  present  and  missing,  a  table 
of  elapsed-time  values  for  certain  specific  operations,  and  subjective  judgments 
based  on  the  experience. 
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1 1 .  DISTRIBUTED  SYSTEMS  DEVELOPMENT  AND 

RUNTIME  SUPPORT  ASSESSORS 


These  assessors  examine  the  ability  of  the  APSE  or  APSE  components  to  sup¬ 
port  software  development  for  distributed  processing  systems,  and  to  provide  runtime  sup¬ 
port  for  distributed  processing  systems. 
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12.  DISTRIBUTED  APSE  ASSESSORS 


These  assessors  examine  the  ability  of  two  or  more  distributed  APSEs  to  com¬ 
municate  in  cooperative  ways  in  supporting  the  development  of  mission  critical  software  at 
diverse  geographical  locations. 
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13. 


“WHOLE  APSE”  ASSESSORS 


These  assessors  examine  or  measure  the  overall  quality  or  performance  of  an 
APSE  considered  as  a  whole  rather  than  as  a  collection  of  individual  parts  individually  as¬ 
sessed.  A  specific  whole-APSE  assessor  may  be  designed  to  achieve  a  limited  objective, 
such  as:  evaluate  the  quality  of  an  APSE  in  supporting  a  team  of  software  developers  per¬ 
forming  a  specific  life-cycle  phase  or  activity  such  as  preliminary  design  or  integration  test¬ 
ing.  The  results  of  such  an  evaluation  could  then  become  one  ingredient  of  an  integrated 
whole-APSE  assessment,  as  described  in  Section  3.3. 


13.1  APSE  CHARACTERIZATION 

Purpose:  The  purpose  of  this  form  is  to  provide  an  overview  or  summary  of  the  capabili¬ 
ties  and  features  of  an  APSE.  This  form  can  be  used  as  an  initial  information  gathering 
device  to  begin  the  process  of  whole-APSE  assessment.  This  information  would  then 
be  supplemented  by  results  of  detailed  evaluations  or  examinations  of  attributes  that  are 
of  specific  interest  to  the  potential  buyer  or  user  of  an  APSE. 

[@RM:  Whole  APSE  Issues  3.,  @RM:  Capacity  6.4.6;  @RM:  Cost  6.4.11; 

®RM:  Maturity  6.4.18;  @RM:  Operability  6.4.20;  @RM:  Power  6.4.21; 

®RM:  Required  Configuration  6.4.26] 

Primary  References: 

Host/OS:  Not  Applicable. 

Vendors/ Agents:  [E&V  Team] 

Method:  Questionnaire 

Inputs:  Blank  APSE  characterization  form  (see  Fig.  13.1-1)  and  APSE  documentation. 

Process: 

1.  Have  APSE  vendor  complete  the  APSE  characterization  form 

2.  Select  APSEs  for  fur. her  investigation  based  on  information  gathered 
from  step  1 . 

Outputs:  Completed  APSE  characterization  form. 
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Name/Acronym: 

Vendor: 

Address: 

Phone  Number: 

Cost  ($,  no  charge,  not  available/applicable) : 

Purchase  _  Seminars  _ 

Maintenance  _  In  House  Classes  _ 

Documentation  _  Educational  Videos  _ 

On-Line  Help  _  On-Line  Tutorials  _ 

Hot-Line  Support _ 

Problem  Reporting/Resolution  Procedures: 

Frequency  of  Updates: 

Usage  Limitations  (License  Restrictions): 

Host/Target(s)  —  Required  Configurations: 

Peripherals  Supported: 

Languages  Supported  &  Interoperability  Features: 

Summary  of  Features: 

Life  Cycle  Support  —  Capabilities/Major  Activity: 

Methodology  Support: 

Manager  ant  Support: 

Application-Specific  Capabilities: 

Documentation  Support  (editors,  word  processors,  document  generators,  desktop  publishing): 


Figure  13,1-1  APSE  Characterization  Form  (Page  1) 
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File/Database/Program  Library  Management  (hierarchical,  relational): 

Access  Control  —  Level  of  Granularity: 

Integration  Mechanism  (standard  file  structures,  database,  standard  intertool  interfaces): 
User  Interface  (command  language,  menus,  icons)  —  Flexibility  vs.  Consistency: 
Extensibility: 

Support  for  Distributed  Development: 

Capacity  (number  of  users,  size  of  project): 

Typical  Usage  Scenarios: 


Developer: 


Production  Process/Vehicles: 

Date  First  Released: 

Previous  Use: 


References  (documentation,  evaluation  results,  case  histories': 


Figure  13.1-1  APSE  Characterization  Form  (Page  2) 
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14.  ADAPTION  ASSESSORS 

These  assessors  examine  the  ease  with  which  an  APSE  or  APSE  component 
can  be  used  beyond  its  original  requirements,  such  as  extending  or  expanding  capabilities 
and  adapting  for  use  in  another  application  or  environment.  This  is  measured  as  the  de¬ 
gree  to  which  this  adaptation  can  be  accomplished  without  reprogramming. 


14.1  HOST  AND  TARGET  CHECKLIST 


Purpose:  Evaluation  of  tools  relative  to  host  and  target  configurations. 

[#RM:  Rehostability  6.4.25; 
rSRM:  Retargetability  6.4.27] 

Primary  References: 

[@Nissen  1984; 

Nissen,  et  al.:  Guidelines  For  Ada  Compiler  Specification  And  Selection:  4.19] 
Vendors/Agents:  [Cambridge  University  Press] 

Method:  Assessment  checklist 

Inputs:  Checklist  and  tool  documentation. 

Process:  Fill  in  the  appropriate  answers  in  the  following  checklist. 


a) 

b) 

c) 

d) 

e) 

f) 


Outputs: 


Host  configuration(s)  required 
Host  operating  system(s)  required 
Target  configuration(s)  supported 
Target  operating  system(s)  supported 
APSE(s)  supported,  if  applicable 
Host-target  communication  supported 

i)  program  loading 

ii)  program  execution  and  debugging. 

A  completed  list  'which  characterizes  the  tool  relative  to  host-target  issues. 


14-1 


E&V  Guidebook,  Version  1.1 


—  SHORTJNTEGER’LAST 
—  LONGJNTEGER'LAST 

•  [@DoD  1983:  3.5.8]  The  values  of: 

—  FLOAT’DIGITS 

—  SHORT_FLOAT’ DIGITS 
—  LONG_FLOAT’DIGITS 

[@DoD  1983:  Names  and  Expressions  4.] 

•  [@DoD  1983:  4.10]  Is  there  a  limit  on  the  range  of  universal 
vcluos  which  exceeds  the  capacity  of  the  compiler 

•  [@DoD  1983:  4.10]  Is  there  a  limit  on  the  accuracy  real  uni¬ 
versal  expressions 

[@DoD  1983:  Tasks  9.] 

•  [@DoD  1983:  9.6]  The  values  of: 

—  DURATION'DELTA 

—  DURATION’SMALL 
—  DURATION’FIRST 
—  DURATION’LAST 

•  [(®DoD  1983:  9.8]  The  values  of: 

—  PRIORITY’FIRST 

—  PRIORITY’LAST 

•  [®DoD  1983:  9.11]  The  restrictions  on  shared  variables 

[raDoD  1983:  Program  Structure  and  Compilation  Issues  10.] 

•  [<aDoD  1983:  10.1]  Initiation,  communication  with,  and  re¬ 
strictions  on  the  main  program 

•  [raDoD  1983:  10.5]  When  tasks  initiated  in  imported  library 
units  will  terminate 

[raDoD  1983:  Exceptions  11.] 

•  [raDoD  1983:  11.1]  Conditions  under  which  these  exceptions 
are  raised: 

—  NUMERIC_ERROR 
—  PROGRAM_ERROR 
—  STORAGE  JERROR 


14-3 


E&V  Guidebook,  Version  1.1 


[@DoD  1983:  Representation  Clauses  and  Implementation-Dependent 
Features  13.] 

•  [(®DoD  1983:  13.4,  F.j  The  list  of  all  restrictions  on  repre¬ 
sentation  clauses 

•  [@DoD  1983:  13.1,  F.]  The  conventions  used  for  any  sys¬ 
tem  generated  name  denoting  system  dependent  components 

•  [@DoD  1983:  13.5,  F.]  The  interpretation  of  expressions  that 
appear  in  address  clauses,  including  those  for  interrupts 

•  [rffiDoD  1983:  13.7]  The  specification  of  package  SYSTEM; 
which  includes  the  values  of: 

—  MINJNT 

—  MAXJNT 

—  MAX_D!GITS 

—  MAX_MANTISSA 

—  FINE_DELTA 

—  TICK 

[@DoD  1983:  13.7.3]  For  a  pre-defined  floating  point  type  F,  the  value 
of: 

—  F’MACHINE_ROUNDS 
—  F'MACHINE_RADIX 
—  F’MACHINE_MANTISSA 
—  F’MACHINE_EMAX 
—  F’MACHINE_EMIN 
—  F’MACHINE_OVERFLOWS 

•  f^aDoD  1983:  13.7.3]  The  values  outside  the  range  of  safe 
numbers  for  real  types 

•  [(5)DoD  1983:  13.10.1}  Any  restriction  on  UN- 
CHECKEDDEALLOCATION 

•  [<®DoD  1983:  13.10.2.  F.]  Any  restriction  on  UN- 
CHECKED_CONVERSION 

[®DoD  1983:  Input-Output  14] 

•  [@DoD  1983:  14.,  F.]  Any  implementation-dependent 
characteristics  of  the 

input-output  packages 
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•  [@DoD  1983:  Implementation-Dependent  Features  F.] 

[@DoD  1983:  F.]  The  name  and  type  of  every  implementa¬ 
tion-dependent  attribute 

Outputs:  A  completed  list  which  characterizes  the  tool  relative  to  machine 
dependencies. 
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99. 


OTHER  ASSESSORS 


This  chapter  contains  instances  of  E&V  technology  that  do  not  conveniently  fit 
one  of  the  earlier  chapters.  It  is  like'y  that  in  future  versions  of  the  Guidebook  some  of 
these  “miscellaneous"  instances  will  be  grouped  together  in  new  chapters,  and  therefore 
moved  out  of  Chapter  99. 


99.1  TEXT  EDITING  CAPABILITIES  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  text  editing  by  developing  a  list  of  functional 
capabilities. 

[@RM:  Text  7. 1.1.1.  @RM:  Power  6.4.21] 

Primary  References: 

[@E&V  Schema  1987:  B.; 

Classificahon  Schema/E&V  Taxonomy  Checklists:  4.4] 

Vendors/Agents:  [E&V  Team] 

Method:  Capabilities  checklist 

Inputs:  Capabilities  checklist  (see  Table  99.1-1)  and  text  editor  documentation. 

Process:  Check  off  capabilities  demonstrated  during  editing  sessions  or  discussed  in 
the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  text  editor. 
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TABLE  99.1-1 

TEXT  EDITING  CAPABILITIES  CHECKLIST 

FEATURE  FOUND 

Locator  Movement 

Left,  Right,  Up,  Down 
Top.  Bottom 
Beginning/End  of  Line 
Next/Previous  Word 
Search/Replace 
Search  Forward 
Search  Backward 
Regular  Expression  Search 
Regular  Expression  Replace 
Multiple  Replace 
Scroll  Up.  Down.  Left.  Right 
Page  Up,  Down 
Buffers 

Copy  Text  To 
Copy  Text  From 
Edit  Multiple  Files 
Split  Screen 
Regions 

Set  Mark 
Delete  Region 
Copy  Region 
Move  Region 
r ile  Manipulation 
Copy  From  File 
Append  To  File 
Macros 

Keyboard  Macros 
Macro  Language 
Language  Sensitive 

Identifiers/Keywords  Search 
Syntax  Templates/Completion 
Reformat  (Pretty  Print) 

Display  High-Level  Structure 
Display  Unclosed  Structures 
Display  Matching  Structures 
Display  Permitted  Constructs 
"Comment  Out"  Code  Block 
Invoke  Syntax  Check/Translation 
LRM  Automated  Access 
Miscellaneous 

Terminal  Independent 
On-Line  Help  Facility 
Minimal  Redisplay  Algorithm 
Key  Redefinition 
Undo  Capability 
Spawn  CLI 

Command  Recall  and  Integration 
Command  Type-Ahead 
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99.2  DATA  BASE  MANAGEMENT  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  data  base  management  by  developing  a  list  of  func¬ 
tional  capabilities. 

[r®RM:  Data  Base  (Object)  Management  7.2. 1.1,  @RM:  Power  6.4.21] 

Primary  References:  [Martin  1986]  D.  Martin,  “Advanced  Database  Techniques,”  MIT 
Press,  1986. 

Vendors/Agents: 

Method:  Capabilities  checklist 

Inputs:  Capabilities  checklist  (see  Table  99.2-1)  and  data  base  manager  documenta¬ 
tion. 

Process.  Check  off  capabilities  demonstrated  by  the  data  base  manager  or  discussed 
in  the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  data  base  manager. 


E&V  Guidebook,  Version  1.1 


TABLE  99.2-1 

DATA  BASE  MANAGEMENT  CAPABILITIES  CHECKLIST 


FEATURE 


Model 

Hierarchical 
Network 
Relational 
Post-Relational 
Object-Oriented 
Data  Dictionary  Management 
Definitions 

Files,  Tables 
Fields,  Attributes 
Relationships 
Subschemas 
Views 

Single-Level 
Multilevel 
Objects.  Entities 
Data  Types 
Subtypes 

User-Defined  Types 
Operations 

Implementation  Parameters 
Nondatabase  Entities 
Listing  Descriptions 
Cross-Referencing  Descriptions 
Dictionary  History 

Automatic  Generation  of  Data  Definition  Statements 
Data  Queries 

Nonprocedural  Language  (4GL) 

Fill-in-the-Form 
Query  by  Example 
Report  Generation 
Query 
Data  Set 
User-Defined 
Update  Mode 
Static 
Dynamic 

Access  Security  Protection 
Access  Control 
By  View 
By  File,  Table 
By  Object,  Entity 
By  Relationship 

By  Operation  (Read,  Write,  Append) 

Data  Encryption 
Dynamic  Password 
Keyword  Input  Protection 
Protection  of  Stored  and  Transmitted  Data 
Referential  Integrity 
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FEATURE 


Access  Conflict  and  Deadlock  Protection 
Access  Locking 

Dynamic  Backout  from  Deadlock 
Undoing  Multiple  Transactions 
Storage 

Full-Length 

Representation  with  Codes 
Data  Packing 
Disk  Space  Management 
Multivolume  Files 
Areas 
File  Groups 

File  Access  Management 
Sequential  File 
Direct  (Random)  Access  File 
Primary  Indexing 
Secondary  Indexing 
Hashing 

Hash-Coded  Index 
Database-Key 

Bit-Vector  Inverted  File  (Bit-Index) 

File  Linking 

One-to-One  Relationship 
One-to-Many  Relationship 
Many-to-Many  Relationship 
Application  Program  Interface 

Application  Development  Language  Interface 

Standard  DBMS  Operations 

Insertion  (Record  Creation  or  Addition) 

Modification  (Field  Update) 

Deletion 

Link  Creation  and  Suppression 
Screen  Generators 

Program/Data  independence  through  Mapping 
Program/Structure  Independence  Using  Multilevel  Views 
Backup  and  Recovery 
Transaction  Logging 
Cold  Restart 
Warm  Restart 

Data  Restructuring  Capabilities  (Views) 

Administration  Capabilities 
Interactive 

Dictionary  Management 
Access  Permission  Management 
Data  Quality  Verification 
Communication  Capabilities 
Import,  Bulk  Data  Loading 
Export,  Flat  File  Conversion 
Single  System  Access 
Multiple  System  Access 
Distributed  Database 
Performance  Monitoring/Tuning 
Miscellaneous 

Terminal  Independent 
On-Line  Help  Facility 
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99.3  ELECTRONIC  MAIL  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  electronic  mail  by  developing  a  list  of  functional 
capabilities. 

[®RM:  Electronic  Mail  7. 2. 1.4,  @RM:  Power  6.4.21] 

Primary  References: 

[@E&V  Schema  1987:  B.; 

Classification  Schema/E&V  Taxonomy  Checklists:  4.4] 

Vendors/Agents:  [E&V  Team] 

Method'  Capabilities  checklist 

Inputs:  Capabilities  checklist  (see  Table  99.3-1)  and  mail  system  documentation. 

Process:  Check  off  capabilities  demonstrated  by  the  mail  system  or  discussed  in  the 
documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  mail  system. 


TABLE  99.3-1 

ELECTRONIC  MAIL  CAPABILITIES  CHECKLIST 


FEATURE 

FOUND 

Send 

Receive 

Immediate  Forwarding 

Immediate  Reply 

Archive 

Print 

Search  For  String 

Edit  Message  To  Be  Sent 

Read  Next  Message 

Read  Previous  Message 

Read  First  Message 

Read  Last  Message 

Position  To  Start  Of  Message 

Keypad  Support 

On-Line  Help  Facility 

Send  To  Distribution  Lists 

Send  Across  Network 

Mail  Filing 
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99.4  REQUIREMENTS  PROTOTYPING  CAPABILITIES  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  requirements  prototyping  by  developing  a  list  of 
functional  capabilities. 

[@RM:  Requirements  Prototyping  7. 3. 2. 2,  @RM:  Power  6.4.21] 

Primary  References: 

Vendors/Agents:  [E&V  Team] 

Method:  Capabilities  checklist 

Inputs:  Capabilities  checklist  (see  Table  99.4-1)  and  requirements  prototyping  docu¬ 
mentation. 

Process:  Check  off  capabilities  demonstrated  by  the  requirements  prototyping  system 
or  discussed  in  the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  requirements  prototyping  system. 


TABLE  99.4-1 

REQUIREMENTS  PROTOTYPING  CAPABILITIES  CHECKLIST 

FEATURE  FOUND 

Standards  Requirements  Libraries 
Executable  Specifications 

Fourth  Generation  Languages  or  Very  High  Lev'’  1  ?nguages 
Reusable  Building  Blocks  and  Associated  Tools 
Man-Machine  Interface  Prototyping  Capabilities 
Applications  Generators 
Previous  Software  Version  Import  Capabilities 
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99.5  PERFORMANCE  MONITORING  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  performance  monitoring  by  developing  a  list  of 
functional  capabilities. 

[@RM:  Performance  Monitoring  7.2.1.10,  tafiM:  Power  6.4.21] 

Primary  References: 

Vendors/Agents:  [E&V  Team] 

Method:  Capabilities  checklist 

Inputs:  Capabilities  checklist  (see  Table  99.5-1)  and  performance  monitor 
documentation. 

Process:  Check  of*  capabilities  demonstrated  by  the  performance  monitor  or  dis 
cussed  in  the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  performance  monitor. 


TABLE  99.5-1 

PERFORMANCE  MONITOR  CAPABILITIES  CHECKLIST 


FEATURE 

FOUND 

Hardware 

CPU  Time  (Real  And  Virtual) 

Memory  Usage 

I/O  Channel  Traffic 

Terminal  Response 

Terminal  Connect  Time 

Terminal  Availability 

Disk  Usage 

Disk  Space  Availability 

Tape  Mounts 

Tape  Drive  Availability 

Printout  Quantity 

Software 

Tool  Usage 

Program  Library  Monitoring 

Wall  Clock  Time 
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99.6  SIMULATION  AND  MODELING  CAPABILITIES  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  simulation  and  modeling  by  developing  a  list  of  func¬ 
tional  capabilities. 

[@RM:  Simulation  and  Modeling  7. 3. 2. 3,  @RM:  Power  6.4.21] 

Primary  References: 

[ISTAR  1987]  Workshop  on  Future  Development  Environments,  Information  Science 
and  Technology  Assessment  for  Research,  Conference  on  Information  Mission 
Area  (IMA)  Productivity,  Department  of  Army  Director  of  Information  Systems  for 
Command,  Control,  Communications  and  Computers.  13-15  April  1987,  pp  28. 

Vendors/ Agents:  [E&V  Team] 

Method:  Capabilities  checklist 

Inputs:  Capabilities  checklist  (see  Table  99.6-1)  and  simulation  and  modeling 
documentation. 

Process:  Check  off  capabilities  demonstrated  by  the  simulation  and  modeling  sys¬ 
tem  or  discussed  in  the  documentation. 

•  Outputs:  A  list  of  capabilities  performed  by  the  simulation  and  modeling  system. 


TABLE  99.6-1 

SIMULATION  AND  MODELING  CAPABILITIES  CHECKLIST 


FEATURE 

FOUND 

Conceptual  Modeling  Support 

Domain-Specific  Knowledge  Base 

Inferencing  Systems 

Operational  Environment  Modeling  Support 

User  Modeling  Support 

Model  Browsers 

Game  and  Risk  Models  Database 

Functional  Allocation  Methodologies  Database 
Scaling  Rules  Database 

Constraint  Evaluation  Tools 

Resource  Utilization  Models 

Precision  Estimators 
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99.7  FILE  MANAGEMENT  CHECKLIST 

Purpose:  Evaluation  of  the  power  of  file  management  by  developing  a  list  of  functional 
capabilities. 

[rSRM:  File  Management  7. 2. 1.3,  @RM:  Power  6.4.21] 

Primary  References: 

[Peterson  1985]  J.L.  Peterson  and  A.  Silberschatz,  "Operating  System  Concepts,"  2nd 
edition,  Addison-Wesley,  1985. 

Vendors,  Agents: 

Method'  Capabilities  checklist 

Inputs:  Capabilities  checklist  (see  Table  99.7-1)  and  file  manager  documentation. 

Process:  Check  off  capabilities  demonstrated  by  the  file  manager  or  discussed  in 
the  documentation. 

Outputs:  A  list  of  capabilities  performed  by  the  file  manager. 
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TABLE  99.7-1 

FILE  MANAGEMENT  CAPABILITIES  CHECKLIST 
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TABLE  99.7-1 

FILE  MANAGEMENT  CAPABILITIES  CHECKLIST  (Continued) 

FEATURE  FOUND 

Allocation  Method 
Contiguous 
Linked 
Indexed 

Access  Management 
Sequential  File 
Direct  (Random)  Access  File 
Primary  Indexing 
Secondary  Indexing 
Hasti-Coded  Indexing 
Access  Security  Protection 

Dynamic  Protection  Structure 
Data  Encryption 
Pile  Password 
Static 
Dynamic 
Multiple 

Access  Control 

Controlled  Operations 
Read 
Write 
Execute 
Append 
Delete 
Access  Matrix 
Global  Table 
Access  List 
Capability  List 
Lock/Key 
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APPENDIX  B 

ACRONYMS  AND  ABBREVIATIONS 


ACEC 

Ada  Compiler  Evaluation  Capability 

ACM 

Association  for  Computing  Machinery 

ACVC 

Ada  Compiler  Validation  Capability 

AES 

Ada  Evaluation  System 

AFB 

Air  Force  Base 

AFWAL 

Air  Force  Wright  Aeronautical  Laboratories 

AIE 

Ada  Integrated  Environment 

AJPO 

Ada  Joint  Program  Office 

ALS 

Ada  Language  System 

ALS/N 

Ada  Language  System/Navy 

ANNA 

Annotation  Language  for  Ada 

ANSI 

American  National  Standards  Institute 

APSE 

Ada  Programming  Support  Environment 

ARTEWG 

Ada  RunTime  Environment  Working  Group  (SIGAda) 

ASCII 

American  Standard  Code  for  Information  Interchange 

ATA 

Advanced  Tactical  Aircraft 

ATF 

Advanced  Tactical  Fighter 

AVF 

Ada  Validation  Facility 

AVO 

Ada  Validation  Office 

BGT 

Benchmark  Generator  Tool 

BSI 

British  Standards  Institute  (UK) 

CAIS 

Common  APSE  Interface  Set 

CIVC 

CAIS  Implementation  Validation  Capability 

CLI 

Command  Language  Instruction 

CMU 

Carnegie  Mellon  University 

CPU 

Central  Processing  Unit 

CSC 

Computer  Software  Component 

DACS 

Data  and  Analysis  Center  for  Software 

DoD 

Department  of  Defense 

DTIC 

Defense  Technical  Information  Center 

EBCDIC 

Extended  Binary  Coded  Decimal  Interchange  Code 

ESD 

Electronic  Systems  Division  (Air  Force) 

E&V 

Evaluation  and  Validation 

GB 

Guidebook 

GIT 

Georgia  Institute  of  Technology 

IBM 

international  Business  Machines  Corporation 

IDA 

Institute  for  Defense  Analysis 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers.  Inc. 
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IPSE 

ISTAR 

I/O 

JIAWG 

KAPSE 

KIT 

KITIA 

LHX 

LRM 

MAPSE 

MCCS 

MIT 

MMI 

MoD 

NBS 

NTIS 

OCD 

OS 

PCTE 

PDL 

PIWG 

RM 

SDI 

SEE 

SEI 

SIGAda 

STARS 

TASC 

UK 

VAX 

VMS 

VSR 

V&V 

WIS 

WWMCCS 


Integrated  Project  Support  Environment 

Information  Science  and  Technology  Assessment  for  Research 

Input/Output 

Joint  Integrated  Avionics  Working  Group 
Kernel  Ada  Programming  Support  Environment 
KAPSE  Interface  Team 

KAPSE  Interface  Team  for  Industry  and  Academia 

Light  Helicopter  Experimental 

Language  Reference  Manual  [@DoD  1983] 

Minimal  Ada  Programming  Support  Environment 
Mission-Critical  Computer  System 
Massachusetts  Institute  of  Technology 
Man-Machine  Interface 
Ministry  of  Defense  (UK) 

National  Bureau  of  Standards 
National  Technical  Information  Service 
Operational  Concept  Document 
Operating  System 

Portable  Common  Tool  Environment 

Program  Design  Language 

Performance  Issues  Working  Group  (SIGAda) 

Reference  Manual 
Strategic  Defense  Initiative 
Software  Engineering  Environment 
Software  Engineering  Institute 

Special  Interest  Group  for  Ada  of  the  Association  for  Computing 
Machinery  (ACM) 

Software  Technology  for  Adaptable,  Reliable  Systems 

The  Analytic  Sciences  Corporation 

United  Kingdom 

Virtual  Address  Extension 

Virtual  Memory  System 

Validation  Summary  Report 

Verification  and  Validation 

WWMCCS  Information  System 

Worldwide  Military  Command  and  Control  System 
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APPENDIX  C 
FORMAL  GRAMMAR 


This  appendix  specifies  sections  of  the  Reference  Manual  and  Guidebook 
(Reference  System)  as  a  formal  grammar.  The  sections  include  chapters  four  through 
seven  of  the  Reference  Manual  (RM),  chapters  four  through  99  of  the  Guidebook  (GB), 
all  explicit  references,  the  tables  of  contents,  the  indices,  and  the  citations.  The  specifica¬ 
tion  is  presented  as  a  partitioned  grammar  for  convenience. 

(The  grammar  is  presented  in  a  modified  Backus-Naur  form.  Brackets  repre¬ 
sent  optionality  when  alone,  and  may  be  marked  by  an  asterisk  "’to  denote  0-N  in¬ 
stances  of  the  production,  or  by  a  sharp  to  denote  1-N  instances.  Angle  brackets 
denote  comments  in  place  of  productions  which  are  too  elaborate  to  express  here.  All 
terminals  of  the  grammar  are  expressed  as  quoted  literals,  or  composite  literals  based  on 
characters  and  character  strings.) 


C.1  FORMAL  REFERENCES 


Throughout  the  Reference  System,  whenever  formal  references  are  made,  a 
single  consistent  set  of  grammar  rules  are  used.  This  includes  reference  from  one  vol¬ 
ume  to  the  other,  reference  from  one  section  in  a  volume  to  another  section  in  the  same 
document,  and  reference  to  documents  outside  the  Reference  System. 

referencejist  ::=  “["  references  references  ]*  "]” 


references 

reference 

phraselist 


::=  reference  [  reference  ]’ 

::=  phrase  [  [  phrase  ] 

[  designator  list  ]] 

[  phrase  ]  designatorjist 

::=  phrase  [  phrase  ]' 
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phrase 

::=  <text  lacking  special  characters> 

designatorjist 

::=  designator  [  designator  ]* 

designator 

lead  ]  lead  [  digits  ]* 

lead 

::=  digits  ]  caps 

digits 

::=  one_to_nine  [zero _to_nine] ' 

onejo^nine 

::= 

zerojonine 

::=  (’O'— ’3*) 

caps 

CA'-'Z’) 

C.2  FORMAL  CHAPTERS 

Those  chapters  of  the  Reference  System  which  are  derived  from  the  classifica¬ 
tion  schema  are  formally  defined  here. 

C.2.1  Chapter  Components 

The  following  rules  define  the  components  which  are  used  to  compose  formal 
chapter  entries. 


header 

:=  designator  phrase 

Prolog 

:=  header  purpose  primary  host 
[vendors_agentsj 

purpose 

:=  "Purpose:”  text 

host 

:=  “Host/OS:”  text 

primary 

:=  “Primary  References:”  referencej 

vendors_agents: 

:=  “Vendors/Agents:”  referencejist 
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method  ::= 

meth_description  inputs 
process  outputs 

meth_description 

::=  “Method:"  text 

inputs  ::= 

“Inputs:”  text 

process 

“Process:"  text 

outputs  ::= 

“Outputs:"  text 

citations  ::= 

“Citations:"  [citations]# 

synopsisjext  ::= 

"Synopsis:"  text 

methods  ::= 

“Methods:”  reference Jist 

text  ::= 

<  prose  text  > 

C.2.2  Chapter  Entries 

Each  numbered  section  of  the  formal  chapters  follows  a  specific  grammar  rule. 
The  following  rules  define  the  format  of  each  chapter  entry. 

synopsi  ::=  header  citations  synopsisjext  [methods] 

E&V  technology  prolog  method 

C.2.3  Formal  Chapter  Ordering 

The  formal  portion  of  the  GB  is  found  in  chapters  four  through  ninety  nine. 

formal  chapters  ::=  [  synopsis  ]* 

[  E&V_technology  ]* 


[  E&VJechnology  ]* 
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C.3  TABLE  OF  CONTENTS 

The  table  of  contents  shares  some  features  with  the  rest  of  the  formal  aspects 

of  the  GB. 

table_of_contents  ::=  (  chapter  ]'  index 
chapter  ::=  designator  phrase  digits 

C.4  CITATIONS 

The  citations  are  found  in  Appendix  A,  and  have  a  formal  structure  as  defined 
in  the  following  grammar.  The  (semantic)  form  of  citation  text  is  taken  from  the  standard 
for  IEEE  Software  Magazine. 


citations 

::=  [  citation  ]' 

citation 

::=  key  body 

key 

::=  “[”  phrase  list  “]” 

body 

reference  list  ]  phrase Jist 
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APPENDIX  D 
VENDORS  AND  AGENTS 

[ARTEWG] 

Mike  Kamrad 

Honeywell  Systems  and  Research  Center 
M/S  MN1 7-2351 
3660  Marshall  St.,  NE 
Minneapolis.  MN  55418 

[BSI,  Milton-Keynes,  UK] 

J.B.  Souter 
BSI  Quality  Assurance 
.  P.O.  Box  375 

Milton  Keynes  MK14  6LL 
UK 

[Cambridge  University  Press] 

Cambridge  University  Press 
32  East  57th  Street 
New  York,  NY  10022 

[Defense  Technical  Information  Center] 

Defense  Technical  Information  Center 
Cameron  Station 
Alexandria,  VA  22314 

[DACS] 

Data  &  Analysis  Center  for  Software 

RADC/COED 

Bldg  101 

Griffiss  AFB,  NY  13441-5700 
Attn:  Document  Ordering 


(612)  782-7321 


0908-220908  x2313 


(703)  274-7633 


(315)  336-0937 
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[E&V  Team] 

Mr.  Raymond  Szymanski 
AFWAL/AAAF 
Wright-Patterson  AFB 
OH  45433-6543 

MILNET:  S2YMANSK@A JPO .SEI.CMU.EDU 
[GIT] 

Georgia  Institute  of  Technology 
Software  Engineering  Research  Center 
Atlanta.  GA  30332-0280 

[MITRE] 

MITRE  Corporation 
Civil  Systems  Division 
7525  Colshire  Drive 
McLean.  VA  22102-3481 

[National  Technical  Information  Service] 
National  Technical  Information  Service 
U.S.  Department  of  Commerce 
5285  Port  Royal  Road 
Springfield,  VA  22161 

[PIWG] 

Dan  M.  Roy 
Ford  Aerospace 
7401 -D  Forbes  Blvd. 

Seabrook,  MD  20706 


(513)  255-2446 
-6730 
AV  785-2446 
-6730 


(404)  894-3180 


(703)  883-6000 


(703)  487-4650 


(301)  464-6800 
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(513)  429-3241 


(412)  268-7700 


(313)  764-1817 


<HJ. S.Government  Printing  Office:  1989  —  648  056/04512 


[SofTech.  Inc.] 

Teresa  L.  Banks 
SofTech,  Inc. 

3100  Presidential  Drive 
Fairborn,  OH  45324-2039 
ARPANET:  HILLM@WPAFB-JALCF 

[SEI] 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

[UMich] 

Russel  M.  Clapp,  Louis  Duchesneau,  Richard  A.  Volz, 

Trevor  N.  Mudge,  and  Timothy  Schultze 

The  Robotics  Research  Laboratory 

The  University  of  Michigan 

Ann  Arbor,  Ml  48109 
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